Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added RepeatAttribute that is aware of RandomizedContext #1125

Merged
merged 10 commits into from
Jan 27, 2025

Conversation

NightOwl888
Copy link
Contributor

  • You've read the Contributor Guide and Code of Conduct.
  • You've included unit or integration tests for your change, where applicable.
  • You've included inline docs for your change, where applicable.
  • There's an open issue for the PR that you are making. If you'd like to propose a change, please open an issue to discuss the change or find an existing issue.

Summary of the changes (Less than 80 chars)

Fixes #1106

Description

This creates a new RepeatAttribute that is nested within LuceneTestCase. In addition to repeating the test X number of times, this attribute ensures that each iteration resets the test seed on RandomizedContext.RandomGenerator so failed test conditions can be replicated without the need to apply the RepeatAttribute.

This sets a precedent for how to test other NUnit extensions (such as attributes) so we can also add tests to confirm they behave as expected. Some bits of the NUnit test approach were added so we can integration test the RepeatAttribute to confirm the correct behavior. The commit cdfd62b includes additional asserts that verify:

  • The same random seed is used for the duration of the test
  • The same test seed is used for the duration of the test
  • The test seed changes from one iteration to the next

Note that RepeatAttribute does not repeat the OneTimeSetUp() or OneTimeTearDown() during each iteration, so every repeated try will use the same culture, time zone, similarity, and codec. Perhaps we can do that at some future point, but it is more complicated to maintain the TestExectutionContext, RandomizedContext and seeds to make that happen.

The format of the random seed string has also changed to use 2 hexadecimal numbers separated by a :. However, existing seeds with one hexidecimal number can still be parsed to replicate test failures. This not only allows us to make features that update the seed, but also will make test replicability more stable across source code updates on tests.

…: Only create an instance of TestResult if it is not already instantiated.
…zedContext aware. Added tests from NUnit to verify it passes RepeatAttribute tests.
…sserts to ensure the random seed and test seed in RandomizedContext are being updated consistently and correctly
…class checking and a test to ensure that we receive the correct result
@NightOwl888 NightOwl888 added the notes:improvement An enhancement to an existing feature label Jan 27, 2025
@NightOwl888 NightOwl888 requested a review from paulirwin January 27, 2025 08:20
Copy link
Contributor

@paulirwin paulirwin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, nicely done. The actual implementation is refreshingly simple! (Although I'm sure the work to get there was not.) I left one comment that could be looked into, but not a must-have change.

@NightOwl888 NightOwl888 merged commit fa630da into apache:master Jan 27, 2025
275 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
notes:improvement An enhancement to an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create [Repeat] attribute that is RandomizedContext aware
2 participants