-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hook the benchmark runner and benchmark visualizations into GitHub #91
base: main
Are you sure you want to change the base?
Conversation
0eb45dc
to
c0b3c3b
Compare
This comment was marked as resolved.
This comment was marked as resolved.
1250336
to
1b92c88
Compare
This comment was marked as resolved.
This comment was marked as resolved.
7bc8dfc
to
beca64a
Compare
This comment was marked as outdated.
This comment was marked as outdated.
0c6401a
to
8994c5c
Compare
This comment was marked as outdated.
This comment was marked as outdated.
8994c5c
to
aee4059
Compare
This comment was marked as outdated.
This comment was marked as outdated.
✔️ Finished running benchmarks from 74b4015. |
This timeout is _much_ lower than what we previously had to use. This is because of the size=fast parameter which cuts down the problem size for benchmarks that take unreasonably long to execute (e.g., SortedLinkedList). It is also an order of magnitude greater than the max execution time using the unthreaded C runtime, so any implementation that times out is unacceptably slow.
Do not push to reactor-c.
74b4015
to
56d9d0a
Compare
✔️ Finished running benchmarks from 56d9d0a. |
Because of security issues with the current setup, all the work in this PR will probably have to be moved to a separate, private repository that is dedicated to continuous benchmarking. Saving benchmark results in a separate repository seems to require a PAT, or at least elevated permissions beyond just GITHUB_TOKEN, which is not recommended. Running actions on a self-hosted runner is also not recommended for private repositories. These observations are based on https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#hardening-for-self-hosted-runners |
The purpose of the other PR, continuous benchmarking, is to catch and prevent unanticipated regressions, as well as to monitor long-term trends in performance.
This PR is intended more as a way to make it as convenient as possible to run benchmarks and to get quick feedback from them while implementing performance optimizations in the runtime. We should have a GitHub Action that satisfies the following:
main
reactor-c
. (Be careful not to merge the branch.)The following tasks are also required, but they might not be worked on until after this PR is merged:
main
)reactor-cpp
reactor-rs
andreactor-ts