Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix statistics for quantification runs and transposition sweeps are misleading #693

Merged
merged 1 commit into from
Sep 11, 2024

Conversation

SSoelvsten
Copy link
Owner

@SSoelvsten SSoelvsten commented Sep 10, 2024

Closes #644

Warning: this is technically a breaking change to the public API. Sorry! But, you didn't use this part anyway... right...? I won't bother upping the version number due to this.

@SSoelvsten SSoelvsten added 🔥 bug Something isn't working ❕ breaking version_number++ labels Sep 10, 2024
@SSoelvsten SSoelvsten self-assigned this Sep 10, 2024
@SSoelvsten SSoelvsten force-pushed the internal/quantify/statistics_fix branch from 9792c6a to 7c70c18 Compare September 10, 2024 10:51
@SSoelvsten SSoelvsten force-pushed the main branch 2 times, most recently from b8ee0fc to 715ae54 Compare September 10, 2024 10:59
Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (QBF 'breakthrough/3x4_19_bwnib')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of -1.15% (stdev: 1.38%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 20847.33 21086.67
Standard Deviation 91.00 290.79

Number of samples: 3

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (QBF 'breakthrough_dual/3x6_10_bwnib')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of -1.01% (stdev: 1.30%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 5161.67 5214.00
Standard Deviation 18.01 67.64

Number of samples: 3

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (QBF 'hex/hein_08_5x5-11_bwnib')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of 0.55% (stdev: 0.56%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 17011.00 16917.33
Standard Deviation 84.15 94.74

Number of samples: 3

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (QBF 'httt/4x4_9_tippy_bwnib')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of -0.14% (stdev: 1.17%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 9481.33 9494.33
Standard Deviation 110.59 60.34

Number of samples: 3

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (Picotrav 'adder')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of -0.14% (stdev: 10.90%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 9629.50 9643.30
Standard Deviation 998.01 1051.13

Number of samples: 10

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (QBF 'ep_dual/8x8_6_e-8-1_p-2-3_bwnib')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of 2.43% (stdev: 2.28%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 5193.00 5067.00
Standard Deviation 118.49 50.21

Number of samples: 3

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (QBF 'connect4/6x6_11_connect4_bwnib')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of 0.45% (stdev: 0.80%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 12246.33 12191.00
Standard Deviation 27.61 98.00

Number of samples: 3

Copy link

codecov bot commented Sep 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.049%. Comparing base (715ae54) to head (cfc7f07).
Report is 1 commits behind head on main.

Additional details and impacted files
@@              Coverage Diff              @@
##              main      #693       +/-   ##
=============================================
+ Coverage   98.040%   98.049%   +0.008%     
=============================================
  Files           93        93               
  Lines         7093      7123       +30     
=============================================
+ Hits          6954      6984       +30     
  Misses         139       139               

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (12-Queens)

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of -0.30% (stdev: 0.65%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 11937.67 11973.67
Standard Deviation 77.02 31.34

Number of samples: 3

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (Picotrav 'arbiter')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of -0.23% (stdev: 16.22%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 68944.88 69102.38
Standard Deviation 10969.02 11208.45

Number of samples: 8

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (QBF 'domineering/5x5_13_bwnib')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of 0.42% (stdev: 0.58%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 15466.33 15401.00
Standard Deviation 58.62 89.60

Number of samples: 3

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (QBF 'ep/8x8_7_e-8-1_p-3-4_bwnib')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of -1.17% (stdev: 2.94%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 29664.33 30010.00
Standard Deviation 109.64 881.94

Number of samples: 3

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (Picotrav 'mem_ctrl')

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of -0.22% (stdev: 0.41%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 112995.00 113238.67
Standard Deviation 215.74 467.89

Number of samples: 3

Copy link

github-actions bot commented Sep 10, 2024

🟡 Regression Test (14-Queens)

'ssoelvsten/adiar/internal/quantify/statistics_fix' is a change in performance of -0.34% (stdev: 0.30%).

... origin/main ssoelvsten/adiar/internal/quantify/statistics_fix
Mean 267375.00 268287.33
Standard Deviation 534.01 807.18

Number of samples: 3

@SSoelvsten SSoelvsten force-pushed the internal/quantify/statistics_fix branch from 7c70c18 to 663b291 Compare September 11, 2024 06:25
…isleading

Warning: this is technically a breaking change to the public API. Sorry! But, you didn't
use this part anyway... right...?
@SSoelvsten SSoelvsten force-pushed the internal/quantify/statistics_fix branch from 663b291 to cfc7f07 Compare September 11, 2024 06:29
@SSoelvsten SSoelvsten merged commit 6097fd4 into main Sep 11, 2024
55 of 57 checks passed
@SSoelvsten SSoelvsten deleted the internal/quantify/statistics_fix branch September 27, 2024 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
❕ breaking version_number++ 🔥 bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Quantification statistics : Total Number of Runs is Incorrect (Singleton Quantification)
1 participant