Skip to content

Commit

Permalink
WIP: TestDriver: Update to v3 (#690)
Browse files Browse the repository at this point in the history
**Action Required:**
We've moved to an authenticated system and you must now provide an
authentication key within our script.

1. Create an account or log into https://app.dashcam.io. 
2. Copy the API key from https://app.dashcam.io/team. You must be the
team "owner"
3. Add the API key as `DASHCAM_API_KEY` within GitHub actions by
following the guide
[here](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions).

This PR moves to our v3 system which includes

- Faster and more stable tests. Notice we're using a `yml` file to guide
the test rather than open prompts. This means the AI will follow the
same steps every time.
- Smarter AI. Better text matching, and more commands. We optimized how
the AI matches text and images, making it way faster and more
fault-tolerant.
- Better summaries. The AI will summarize the full test execution as
well as errors
- Assertions. We can now explicitly ask the AI to verify things on
screen
- Embedded tests. We can nest scripts for reusable components.

---------

Co-authored-by: orliesaurus <orliesaurus@users.noreply.github.com>
  • Loading branch information
ianjennings and orliesaurus authored Jul 2, 2024
1 parent d54a616 commit 3d72e97
Show file tree
Hide file tree
Showing 3 changed files with 63 additions and 68 deletions.
84 changes: 34 additions & 50 deletions .github/workflows/regression.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: TestDriver.ai Regression Testing
name: TestDriver.ai Regression Testing - Waveterm
on:
push:
branches:
Expand All @@ -14,53 +14,37 @@ permissions:
contents: read # To allow the action to read repository contents
pull-requests: write # To allow the action to create/update pull request comments


jobs:
test:
name: TestDriver
runs-on: ubuntu-latest
steps:
- uses: dashcamio/testdriver@main
id: testdriver
with:
version: v2.12.12
prerun: |
rm ~/Desktop/WITH-LOVE-FROM-AMERICA.txt
cd ~/actions-runner/_work/testdriver/testdriver/
brew install go
brew tap scripthaus-dev/scripthaus
brew install corepack
brew install scripthaus
corepack enable
yarn install
scripthaus run build-backend
echo "Yarn"
yarn
echo "Rebuild"
scripthaus run electron-rebuild
echo "Webpack"
scripthaus run webpack-build
echo "Starting Electron"
scripthaus run electron 1>/dev/null 2>&1 &
echo "Electron Done"
exit
prompt: |
1. wait 10 seconds
1. click "Get Started"
1. validate that overlapping text does not appear in the application
1. focus the Wave input with the keyboard shorcut Command + I
1. type 'ls' into the input
1. press return
1. validate Wave shows the result of 'ls'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- uses: peter-evans/create-or-update-comment@v4
if: ${{always()}}
with:
issue-number: ${{ github.event.pull_request.number }}
body: |
## TestDriver Summary
${{ steps.testdriver.outputs.markdown }}
${{ steps.testdriver.outputs.summary }}
reactions: |
+1
-1
test:
name: "TestDriver"
runs-on: ubuntu-latest
steps:
- uses: dashcamio/testdriver@main
id: testdriver
with:
version: v3.5.6
key: ${{secrets.DASHCAM_API}}
prerun: |
cd ~/actions-runner/_work/testdriver/testdriver/
brew install go
brew tap scripthaus-dev/scripthaus
brew install corepack
brew install scripthaus
corepack enable
yarn install
scripthaus run build-backend
echo "Yarn"
yarn
echo "Rebuild"
scripthaus run electron-rebuild
echo "Webpack"
scripthaus run webpack-build
echo "Starting Electron"
scripthaus run electron 1>/dev/null 2>&1 &
echo "Electron Done"
cd /Users/ec2-user/Downloads/td/
npm rebuild
exit
prompt: |
1. /run /Users/ec2-user/actions-runner/_work/testdriver/testdriver/.testdriver/wave1.yml
18 changes: 0 additions & 18 deletions .testdriver/prerun.sh

This file was deleted.

29 changes: 29 additions & 0 deletions .testdriver/wave1.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
steps:
- step:
- command: focus-application
name: Electron
- command: hover-text
description: Get Started
action: click
- command: hover-text
description: Settings
action: click
- command: hover-text
description: 13px
action: click
- command: hover-text
description: 12px
action: click
- command: hover-text
description: Dark
action: click
- command: hover-text
description: Light
action: click
- command: hover-text
description: workspace-1
action: click
- step:
- command: assert
description: the terminal is white

0 comments on commit 3d72e97

Please sign in to comment.