Test sharding - take 2 #2832
Merged
Test sharding - take 2 #2832
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
iwahbe
force-pushed
the
iwahbe/test_sharding_wip
branch
4 times, most recently
from
35b8d6f to
dfeb591
Compare
While github.com/pulumi/sharder should be optimal, github.com/pulumi/shard should approach optimality as the number of tests grows. I want to see the degradation in test performance we see when using a simpler solution.
iwahbe
force-pushed
the
iwahbe/test_sharding_wip
branch
from
dfeb591 to
3b8c80d
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2832 +/- ##
==========================================
- Coverage 68.69% 67.48% -1.21%
==========================================
Files 325 325
Lines 41621 41621
==========================================
- Hits 28593 28090 -503
- Misses 11422 11953 +531
+ Partials 1606 1578 -28 ☔ View full report in Codecov by Sentry. |
VenelinMartinov
approved these changes
Jan 14, 2025
This test flakes in the new sharded CI. It wasn't run before (`x/` was ignored in testing), so I'm not worrying about it. It only tests old and unmaintained code at this point.
iwahbe
force-pushed
the
iwahbe/test_sharding_wip
branch
from
0fc8d8a to
bfec06a
Compare
This function has caused repeated test failures with concurrent map mutations, the most recent of which is https://github.com/pulumi/pulumi-terraform-bridge/actions/runs/12772071950/job/35600764628?pr=2832. This commit converts the function to return a copy instead of mutating.
iwahbe
force-pushed
the
iwahbe/test_sharding_wip
branch
from
bfec06a to
a913512
Compare
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Follow-up work on #2810.
This PR aimed to validate the more complex test optimization strategy of #2810 (pre-calculated perfect allocation; implemented by github.com/pulumi/sharder) by demonstrating the effects of a simpler strategy (random allocation; implemented by github.com/pulumi/shard).
Initial testing showed that both strategies were bounded by the 15 minute job
Test and Lint / test (1.22.x, windows-latest, DEFAULT, 0), which is not effected by sharding.Since both approaches appear identical, we will merge the simpler strategy.