Test sharding

[VenelinMartinov]

This PR implements a test sharding strategy to speed up the bridge tests from 1 hour 30 minutes to under 15 minutes.

It uses https://github.com/VenelinMartinov/sharder which is a package for generating regexes to plug into go test based on the test profile data. The script is based on the previous work by @blampe in https://github.com/pulumi/shard.

This PR also adds two new make commands:

• make test_profile this regenerates the test profile data, so that shard partitioning can be more fair
• make test_shard. This is only run in CI. This command picks up the SHARD_CMD env var to determine which tests to run. SHARD_CMD is populated in CI by the sharder package and contains a regex which should match the correct tests to run.

Sharding is only done for ubuntu tests as they take the longest.

Example run: https://github.com/pulumi/pulumi-terraform-bridge/actions/runs/12656374664

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.69%. Comparing base (f68bff0) to head (33505e7).
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2810      +/-   ##
==========================================
- Coverage   68.69%   68.69%   -0.01%     
==========================================
  Files         325      325              
  Lines       41621    41621              
==========================================
- Hits        28593    28590       -3     
- Misses      11422    11424       +2     
- Partials     1606     1607       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[VenelinMartinov]

This change is part of the following stack:

• move providerbuilder to pkg/pf/internal #2804
  • Add pb.Provider.ToProviderInfo test helper method #2805
    • Remove bridgedProvider test utilities in favour of ToProviderInfo #2812
      • Move PF detailed diff test to separate folder #2813
        • Refactor PF detailed diff tests #2814
          • Refactor sdkv2 detailed diff tests #2816
            • Test sharding #2810 ◀
              • SDKv2 Diff tests for primitive types #2829
                • SDKv2 Diff tests for Map #2830

_{Change managed by git-spice.}

[iwahbe]

If I add new tests, do I need to re-run make test_profile? If so, that means that I need to re-run the entire test suite on my laptop to add a single test (which is a major usability regression). If not, then I don't understand how the test profile gets updated.

github.com/pulumi/sharder's REAMDE is very sparse, which doesn't give me confidence. I'd like to be able to read the README and understand how the tool works, not just a short explanation of what it does. The README should also explain the irreconcilable differences between github.com/pulumi/sharder and github.com/pulumi/shard.

[VenelinMartinov]

@iwahbe I've updated the README on sharder. Let me know if that is now clearer.

If I add new tests, do I need to re-run make test_profile?

No, that is not necessary. I've added a description to the sharder README about it. The TLDR is that the last shard will run all tests not run by any previous ones. This means that if the test timings are not updated the shards might become unbalanced but no tests will be skipped.

In practice, I think this will mean that we'll re-record them once we add a good chunk of tests but don't need to touch that otherwise.

[iwahbe]

I'm testing how much performance we gain with this method vs just using pulumi/shard. If the gain is significant, then I'll approve.

[iwahbe]

Closing in favor of #2832, which was stacked on top of this PR.