Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs for Circleci rerun failed tests feature #146

Merged
merged 14 commits into from
Dec 4, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docusaurus/docs/knapsack_pro-ruby/guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,8 @@ jobs:

Here you can find an example of a [Rails project config on CircleCI 2.0](https://docs.knapsackpro.com/2017/circleci-2-0-capybara-feature-specs-selenium-webdriver-with-chrome-headless).

See how to configure [advanced CircleCI features](../../ruby/circleci.md) with Knapsack Pro to collect reports and rerun failed tests.
ArturT marked this conversation as resolved.
Show resolved Hide resolved

</ShowIfSearchParamAndValue>

<ShowIfSearchParamAndValue searchParam="ci" value="cirrus-ci">
Expand Down
45 changes: 44 additions & 1 deletion docusaurus/docs/ruby/circleci.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ end
name: RSpec with Knapsack Pro in Queue Mode
command: |
export CIRCLE_TEST_REPORTS=/tmp/test-results
mkdir $CIRCLE_TEST_REPORTS
mkdir -p $CIRCLE_TEST_REPORTS
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"

- store_test_results:
Expand All @@ -62,3 +62,46 @@ end
</Tabs>

You can find a complete CircleCI configuration in [RSpec testing parallel jobs with CircleCI and JUnit XML report](https://docs.knapsackpro.com/2021/rspec-testing-parallel-jobs-with-circleci-and-junit-xml-report).

## Rerun failed tests
ArturT marked this conversation as resolved.
Show resolved Hide resolved

Use the [CircleCI rerun failed tests](https://circleci.com/docs/rerun-failed-tests/) feature with Knapsack Pro to only rerun a subset of tests instead of rerunning the entire test suite when a transient test failure arises.

Thanks to that, you save time and CircleCI resources. See a [video demo](https://www.youtube.com/watch?v=gf2BK_ZukUg).
ArturT marked this conversation as resolved.
Show resolved Hide resolved

```yaml title=".circleci/config.yml"
- run:
name: RSpec with Knapsack Pro in Queue Mode
command: |
export CIRCLE_TEST_REPORTS=/tmp/test-results
mkdir -p $CIRCLE_TEST_REPORTS

# split slow spec files by test examples
ArturT marked this conversation as resolved.
Show resolved Hide resolved
# https://docs.knapsackpro.com/ruby/split-by-test-examples/
export KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true

# highlight-start
# Use circleci CLI to find out if we need to run all tests or rerun failed tests
# We are telling circleci to split the tests across 1 node to get the full list of all tests for consideration. We leave the splitting to Knapsack Pro.
ArturT marked this conversation as resolved.
Show resolved Hide resolved
circleci tests glob "spec/**/*_spec.rb" | circleci tests run --index 0 --total 1 --command ">/tmp/tests_to_run.txt xargs echo" --verbose > /tmp/tests_to_run.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that with -n1 you won't need the sed later:

Suggested change
circleci tests glob "spec/**/*_spec.rb" | circleci tests run --index 0 --total 1 --command ">/tmp/tests_to_run.txt xargs echo" --verbose > /tmp/tests_to_run.txt
circleci tests glob "spec/**/*_spec.rb" | circleci tests run --index 0 --total 1 --command ">/tmp/tests_to_run.txt xargs -n1 echo" --verbose > /tmp/tests_to_run.txt

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works. Cool. 🚀


# replace all spaces with newlines in the file
sed -i 's/ /\n/g' /tmp/tests_to_run.txt

# tell Knapsack Pro to run only tests from the file (and still use Knapsack Pro Queue Mode magic)
ArturT marked this conversation as resolved.
Show resolved Hide resolved
if [[ -s "/tmp/tests_to_run.txt" ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is checking if the file exists and is non-empty. Should that be always the case?

What if we remove the if and let Knapsack Pro raise an error?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point.

export KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE=/tmp/tests_to_run.txt
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"
fi
# highlight-end

- store_test_results:
path: /tmp/test-results

- store_artifacts:
path: /tmp/test-results
destination: test-results
```

- We use [`KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE`](reference.md#knapsack_pro_test_file_list_source_file) to feed Knapsack Pro with a list of test files to run.
- Optionally, you can use [`KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES`](split-by-test-examples.md) to split slow test files by test examples automatically.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd either mention both in the snippet, or here. Currently, KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES is present in both.

If the links in the snippet cannot be made clickable, I'd leave them both down here.

Suggested change
- We use [`KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE`](reference.md#knapsack_pro_test_file_list_source_file) to feed Knapsack Pro with a list of test files to run.
- Optionally, you can use [`KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES`](split-by-test-examples.md) to split slow test files by test examples automatically.
The snippet above uses:
- [`KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES`] to split tests by examples
- [`KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE`] to specify what tests to run