Allow flaky-rate and failure-rate with historical

[robfischer79]

Hi,

I pretty much like this approach. It is very useful to show my test results in a nice way. However, I have a lot of tests as well as CI runs.
I think it would be a great feature if the historical command, which already takes care of downloading the results, could be enriched with the flaky and failure rates as a nice health check of your project. Or is there any other way it can be combined without downloading results again and by restricting to the exact same artifacts?

Many thanks,

Rob

[Ma11hewThomas]

Hi, I'm glad you like the tool, I can imagine this is frustrating.

The best solution is to use the custom report and --fetch-previous-results arg - for example -

npx github-actions-ctrf custom ctrf/ctrf-report.json templates/custom-report.hbs --fetch-previous-results`

The custom report templates/custom-report.hbs should include all three reports you need and look something like below:

| **Build {{getCtrfEmoji "build"}}** | **Result {{getCtrfEmoji "result"}}** | **Tests {{getCtrfEmoji "tests"}}** | **Passed {{getCtrfEmoji "passed"}}** | **Failed {{getCtrfEmoji "failed"}}** | **Skipped {{getCtrfEmoji "skipped"}}** | **Pending {{getCtrfEmoji "pending"}}** | **Other {{getCtrfEmoji "other"}}** | **Flaky {{getCtrfEmoji "flaky"}}** | **Duration {{getCtrfEmoji "duration"}}** |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [#{{ctrf.environment.extra.runNumber}}]({{ctrf.environment.extra.buildUrl}}) | {{getCtrfEmoji ctrf.summary.extra.result}} | {{ctrf.summary.tests}} | {{ctrf.summary.passed}} | {{ctrf.summary.failed}} | {{ctrf.summary.skipped}} | {{ctrf.summary.pending}} | {{ctrf.summary.other}} | {{countFlaky ctrf.tests}} | {{formatDuration ctrf.summary.start ctrf.summary.stop}} |
{{#each ctrf.extra.previousReports}}
| [#{{this.results.environment.extra.runNumber}}]({{this.results.environment.extra.buildUrl}}) | {{getCtrfEmoji this.results.summary.extra.result}} | {{this.results.summary.tests}} | {{this.results.summary.passed}} | {{this.results.summary.failed}} | {{this.results.summary.skipped}} | {{this.results.summary.pending}} | {{this.results.summary.other}} | {{this.results.summary.extra.flaky}} | {{formatDuration this.results.summary.start this.results.summary.stop}} |
{{/each}}

#### Overall Flaky Rate: {{formatRate ctrf.summary.extra.flakyRate 2}}%

| Test 📝 | Attempts 🎯 | Pass ✅ | Fail ❌ | Flaky Rate % 🍂 |
| --- | --- | --- | --- | --- |
{{#each (sortTestsByFlakyRate ctrf.tests)}}
| {{name}} | {{extra.totalAttempts}} | {{extra.passedCount}} | {{extra.failedCount}} | {{formatRate extra.flakyRate 2}} |
{{/each}}

<sub><i>Measured over {{ctrf.summary.extra.reportsUsed}} runs.</i></sub>

#### Overall Fail Rate: {{formatRate ctrf.summary.extra.failRate 2}}%

| Test 📝 | Runs 🎯 | Pass ✅ | Fail ❌ | Fail Rate % | 
| --- | --- | --- | --- | --- | 
{{#each (sortTestsByFailRate ctrf.tests)}}
| {{name}} | {{extra.finalResults}} | {{extra.passedCount}} | {{extra.finalFailures}} | {{formatRate extra.failRate 2}} |
{{/each}}

<sub><i>Measured over {{ctrf.summary.extra.reportsUsed}} runs.</i></sub>

[robfischer79]

Oh, that is nice. Seems I overlooked that argument. And not frustrating at all, I just thought it might be a nice feature you would be interested in as well maybe. Thank you a lot, I try it out!