Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add pr-message #41

Merged
merged 3 commits into from
Oct 16, 2024
Merged

fix: add pr-message #41

merged 3 commits into from
Oct 16, 2024

Conversation

Ma11hewThomas
Copy link
Contributor

No description provided.

Copy link

With PR Comment - Run #22

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

Copy link

With PR Comment No Fails - Run #22

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 10 0 0 0 0 0 00:00:11

🎉 All tests passed!

Github Actions Test Reporter CTRF

Copy link

With PR Comment On Fail Only With Fails - Run #22

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

Copy link

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3
Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ �[31mTimed out 5000ms waiting for �[39m�[2mexpect(�[22m�[31mlocator�[39m�[2m).�[22mtoHaveTitle�[2m(�[22m�[32mexpected�[39m�[2m)�[22m

Locator: locator(':root')
Expected pattern: �[32m/Playwrc cight/�[39m
Received string: �[31m"Fast and reliable end-to-end testing for modern web apps | Playwright"�[39m
Call log:
�[2m- expect.toHaveTitle with timeout 5000ms�[22m
�[2m - waiting for locator(':root')�[22m
�[2m - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>�[22m
�[2m - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"�[22m
|
| should fail to update profile on network failure | failed ❌ | Network Timeout |
| should fail to update profile on network failure | failed ❌ | No failure message |

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

  • Passed all unit tests
  • Review code quality
  • Update documentation

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
Name Status Failure Message
should be able to login passed No failure message
should display title failed Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
    |
    | should be able to update profile | passed | No failure message |
    | should be able to logout | skipped | No failure message |
    | should validate user settings | passed | No failure message |
    | should fail to update profile on network failure | failed | Network Timeout |
    | should fail to update profile on network failure | failed | No failure message |
    | should load user data | pending | No failure message |
    | should handle session timeouts | passed | No failure message |
    | should clean up user session on logout | other | No failure message |
    | should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

Copy link

With PR Comment - Run #24

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

Copy link

With PR Comment No Fails - Run #24

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 10 0 0 0 0 0 00:00:11

🎉 All tests passed!

Github Actions Test Reporter CTRF

Copy link

With PR Comment On Fail Only With Fails - Run #24

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

Copy link

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3
Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ �[31mTimed out 5000ms waiting for �[39m�[2mexpect(�[22m�[31mlocator�[39m�[2m).�[22mtoHaveTitle�[2m(�[22m�[32mexpected�[39m�[2m)�[22m

Locator: locator(':root')
Expected pattern: �[32m/Playwrc cight/�[39m
Received string: �[31m"Fast and reliable end-to-end testing for modern web apps | Playwright"�[39m
Call log:
�[2m- expect.toHaveTitle with timeout 5000ms�[22m
�[2m - waiting for locator(':root')�[22m
�[2m - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>�[22m
�[2m - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"�[22m
|
| should fail to update profile on network failure | failed ❌ | Network Timeout |
| should fail to update profile on network failure | failed ❌ | No failure message |

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

  • Passed all unit tests
  • Review code quality
  • Update documentation

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
Name Status Failure Message
should be able to login passed No failure message
should display title failed Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
    |
    | should be able to update profile | passed | No failure message |
    | should be able to logout | skipped | No failure message |
    | should validate user settings | passed | No failure message |
    | should fail to update profile on network failure | failed | Network Timeout |
    | should fail to update profile on network failure | failed | No failure message |
    | should load user data | pending | No failure message |
    | should handle session timeouts | passed | No failure message |
    | should clean up user session on logout | other | No failure message |
    | should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

Copy link

Pull Request Arg

Tests 📝Passed ✅Failed ❌Skipped ⏭️Pending ⏳Other ❓Flaky 🍂Duration ⏱️
1053111300:00:11
Github Actions Test Reporter CTRF

Failed Tests

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message
Github Actions Test Reporter CTRF

Flaky Tests

NameStatusRetriesFlaky 🍂
should be able to update profilepassed ✅2🍂
should handle session timeoutspassed ✅1🍂
should allow user to change passwordpassed ✅3🍂
Github Actions Test Reporter CTRF

Detailed Test Results

NameStatusmsFlaky 🍂
should be able to loginpassed ✅1200
should display titlefailed ❌800
should be able to update profilepassed ✅1200Yes
should be able to logoutskipped ⏭️0
should validate user settingspassed ✅1100
should fail to update profile on network failurefailed ❌900
should fail to update profile on network failurefailed ❌900
should load user datapending ⏳0
should handle session timeoutspassed ✅950Yes
should clean up user session on logoutother ❓1050
should allow user to change passwordpassed ✅1300Yes
Github Actions Test Reporter CTRF

Copy link

With PR Comment - Run #26

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

Copy link

With PR Comment No Fails - Run #26

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 10 0 0 0 0 0 00:00:11

🎉 All tests passed!

Github Actions Test Reporter CTRF

Copy link

With PR Comment On Fail Only With Fails - Run #26

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

Copy link

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3
Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ �[31mTimed out 5000ms waiting for �[39m�[2mexpect(�[22m�[31mlocator�[39m�[2m).�[22mtoHaveTitle�[2m(�[22m�[32mexpected�[39m�[2m)�[22m

Locator: locator(':root')
Expected pattern: �[32m/Playwrc cight/�[39m
Received string: �[31m"Fast and reliable end-to-end testing for modern web apps | Playwright"�[39m
Call log:
�[2m- expect.toHaveTitle with timeout 5000ms�[22m
�[2m - waiting for locator(':root')�[22m
�[2m - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>�[22m
�[2m - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"�[22m
|
| should fail to update profile on network failure | failed ❌ | Network Timeout |
| should fail to update profile on network failure | failed ❌ | No failure message |

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

  • Passed all unit tests
  • Review code quality
  • Update documentation

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
Name Status Failure Message
should be able to login passed No failure message
should display title failed Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
    |
    | should be able to update profile | passed | No failure message |
    | should be able to logout | skipped | No failure message |
    | should validate user settings | passed | No failure message |
    | should fail to update profile on network failure | failed | Network Timeout |
    | should fail to update profile on network failure | failed | No failure message |
    | should load user data | pending | No failure message |
    | should handle session timeouts | passed | No failure message |
    | should clean up user session on logout | other | No failure message |
    | should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

Copy link

Pull Request Arg

Tests 📝Passed ✅Failed ❌Skipped ⏭️Pending ⏳Other ❓Flaky 🍂Duration ⏱️
1053111300:00:11
Github Actions Test Reporter CTRF

Failed Tests

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

  • expect.toHaveTitle with timeout 5000ms
  • waiting for locator(':root')
  • locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…
  • unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message
Github Actions Test Reporter CTRF

Flaky Tests

NameStatusRetriesFlaky 🍂
should be able to update profilepassed ✅2🍂
should handle session timeoutspassed ✅1🍂
should allow user to change passwordpassed ✅3🍂
Github Actions Test Reporter CTRF

Detailed Test Results

NameStatusmsFlaky 🍂
should be able to loginpassed ✅1200
should display titlefailed ❌800
should be able to update profilepassed ✅1200Yes
should be able to logoutskipped ⏭️0
should validate user settingspassed ✅1100
should fail to update profile on network failurefailed ❌900
should fail to update profile on network failurefailed ❌900
should load user datapending ⏳0
should handle session timeoutspassed ✅950Yes
should clean up user session on logoutother ❓1050
should allow user to change passwordpassed ✅1300Yes
Github Actions Test Reporter CTRF

@Ma11hewThomas Ma11hewThomas merged commit 5782d67 into main Oct 16, 2024
18 checks passed
@Ma11hewThomas Ma11hewThomas deleted the fix/pr-message branch December 7, 2024 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant