Implement Test API #12935

tsmaeder · 2023-09-24T04:17:55Z

What it does

Implements support for the VS Code testing API in the usual fashion: there is a test service interface and implementation that is plugin-agnostic (packages/test?) and a plugin-specific implementation in the plugin-ext package.

How to test

There is a sample test controller contribution active in the sample applications via the "api-samples" folder. The test controller shows up in the Test Explorer View. You'll need to invoke the "Add Some Tests" command to populate the controller with tests.
In addition, I used the https://open-vsx.org/extension/ms-playwright/playwright extension to test the code on the Theia source code itself. On Windows, you'll need to fix the playwright "theia:start" yarn task like so:

rimraf .tmp.cfg && set THEIA_CONFIG_DIR=./.tmp.cfg  && yarn --cwd ../browser start

Note that this is minimal implementation that lets us satisfy the test API. It consists of "Test Exlorer" and "Test Result" views showing up in the left sidebar and a "Test Output" and a "Test Result" view showing up at the bottom. You'll have to open those second two views manually.

Follow-ups

While the API is complete, there are a couple of things which the UI will never use:

No support for "continous run".
Even if a test plugin supports continoous runs, the UI will never invoke a test run in this fashion.
No support for invalidateTestResults.
The UI has no support for showing test results as outdated, so that API has no function
No Test Markers
We show no marker annotations for tests in the editor. To run tests, you'll need to use the UI in the Test Explorer view.
Build out convenience functions in the UI, like
- automatically open "Test Output" and "Test Result" views on double click
- be smarter on what to show in the output and results: aggregate data if non available on the selection
No support for "coverage" test profiles
Since the whole coverage API is in "proposed" state, it is not implemented.

Review checklist

As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

As a reviewer, I agree to behave in accordance with the review guidelines

tsmaeder · 2023-09-24T04:26:29Z

2023-09-24.06-23-23.mp4

msujew · 2023-09-25T11:04:01Z

@tsmaeder Quid pro quo offer: You'll review #12618 and I'll take a look at this PR? :)

tsmaeder · 2023-09-25T11:12:59Z

@msujew Can do, but not before next week.

msujew · 2023-09-25T11:20:26Z

Sure, I'll take what I get :D

I'll probably already review your PR this week, since I'll be on vacation until EclipseCon starting Friday.

examples/api-samples/package.json

examples/browser/package.json

packages/test/src/browser/view/test-result-widget.ts

packages/test/src/browser/view/test-run-view-contribution.ts

packages/test/src/browser/view/test-run-widget.tsx

packages/test/src/browser/view/test-execution-state-manager.ts

packages/test/src/browser/view/test-output-ui-model.ts

packages/test/src/browser/view/test-output-widget.ts

packages/test/src/browser/view/test-run-widget.tsx

tsmaeder · 2023-10-24T06:54:58Z

@msujew could I get another review-pass? Thursday is release day ;-)

msujew

Alright, code and functionality look pretty good to me. There are a few things we can probably improve on in the UX department, but this is good to go for a first release.

Currently, all command buttons are shown for every test item which leads to a lot of UI clutter. VSCode only shows the button on the currently selected tree item.
VSCode shows the test duration of all tests (similar to the duration of the execution of notebook cells)
During the discovery of tests, VSCode shows a loading icon on related tree elements. This is currently missing.
VSCode has dedicated testing theme colors. We are currently using colors from other areas of vscode - which works good enough for now.

I have also left a few minor comments on the code that were only visible on manual inspection of the app.

msujew · 2023-10-24T09:40:38Z

packages/test/src/browser/view/test-view-contribution.ts

+    export const RUN_TEST_WITH_PROFILE: Command = Command.toDefaultLocalizedCommand({
+        id: TestCommandId.RunUsingProfileAction,
+        category: 'Test',
+        label: 'Execute using Profile... '


Minor: Too much whitespace, translation fails due to that.

Suggested change

label: 'Execute using Profile... '

label: 'Execute using Profile...'

msujew · 2023-10-24T09:45:10Z

tsconfig.json

@@ -1,6 +1,6 @@
 {
  "extends": "./configs/base.tsconfig.json",
-  "include": [],
+  "include": [ ],


Nitpick:

Suggested change

"include": [ ],

"include": [],

The Theia workspace preferences have "format on save" enabled and this is what the formatter gives when running Theia blueprint. If that is not correct we should either fix the preference of fix whatever bug is causing that in Theia. Part of the "self-hosting" epic?

msujew · 2023-10-24T09:52:21Z

packages/test/src/browser/view/test-view-contribution.ts

+                if (TestItem.is(t)) {
+                    this.fileSystem.resolve(t.uri!).then(stat => {
+                        if (stat.isFile) {
+                            this.navigationService.reveal(NavigationLocation.create(t.uri!, t.range ? t.range.start : { line: 0, character: 0 }));


Suggestion: This range isn't calculated correctly. It seems like the test extensions return range information in the monaco format, where files start at line 1, column 1. But we assume that files start at line 0, column 0, leading to an incorrect offset when executing the command.

To return ranges in Monaco format would be a clear mistake on the part of the extensions: https://code.visualstudio.com/api/references/vscode-api#Position clearly states 0-based indices. There might be a mistake in the subsequent computations, though 🤷

Might be, but the behavior is clearly different between vscode and Theia - especially since Theia doesn't place the cursor at the start of the line but after the first character. It might be that we accidentally convert them to the monaco format.

tsmaeder · 2023-10-24T11:32:47Z

* Currently, all command buttons are shown for every test item which leads to a lot of UI clutter. VSCode only shows the button on the currently selected tree item.

We were missing a bit of css. FIxed.

* VSCode shows the test duration of all tests (similar to the duration of the execution of notebook cells)

* During the discovery of tests, VSCode shows a loading icon on related tree elements. This is currently missing.

* VSCode has dedicated [testing theme colors](https://github.com/microsoft/vscode/blob/main/src/vs/workbench/contrib/testing/browser/theme.ts). We are currently using colors from other areas of vscode - which works good enough for now.

This PR is by no means meant to be full copy of the VS Code testing UI. Minimal, but functional is the goal.

msujew · 2023-10-24T11:47:57Z

@tsmaeder All good, these were just remarks for future work, the PR itself is ready to be merged in my view 👍 (just so people interested in improving stuff have a starting point)

contributed on behalf of STMicroelectronics Signed-off-by: Thomas Mäder <t.s.maeder@gmail.com>

Add Testing panel Start the initial tree structure. (same structure as Test API) No label is shown, no active refresh. Basic css implementation

…t API Project lead: Kevin Clapperton Developers: Zakaria Diabi, Mathieu Bussières Testers: Kevin Clapperton, Théo St-Denis Signed-off-by: Zakaria Diabi <d.zaki2396@gmail.com> Signed-off-by: Mathieu Bussières <mathblin@hotmail.com> Signed-off-by: Théo St-Denis <theo.st-denis@polymtl.ca> Signed-off-by: Kevin Clapperton <keclapperton@gmail.com> Co-authored-by: Mathieu Bussières <mathblin@hotmail.com> Co-authored-by: Théo St-Denis <theo.st-denis@polymtl.ca> Co-authored-by: Kevin Clapperton <keclapperton@gmail.com>

- Adapted plugin-side services to work with Theia test service - Added views for test runs, results and output - Built out Test Explorer view - Added various commands to run tests and manage results The result is a minimal, but functional UI for working with tests Fixes eclipse-theia#10669 Contributed on behalf of STMicroelectronics Signed-off-by: Thomas Mäder <t.s.maeder@gmail.com>

tsmaeder requested review from jonah-iden, msujew and planger September 24, 2023 04:18

msujew added the vscode issues related to VSCode compatibility label Sep 24, 2023

This was referenced Sep 26, 2023

vscode - Stub TestController invalidateTestResults eclipsesource/theia#92

Closed

[vscode] Stub TestController invalidateTestResults #12944

Merged

tsmaeder mentioned this pull request Sep 29, 2023

Tests API implementation #12175

Closed

1 task

msujew reviewed Oct 2, 2023

View reviewed changes

tsmaeder force-pushed the 10669_test_api branch from 3e396ad to 31c3806 Compare October 4, 2023 17:55

tsmaeder requested a review from msujew October 24, 2023 06:43

tsmaeder force-pushed the 10669_test_api branch from bcfbe18 to 0f3bba1 Compare October 24, 2023 08:55

msujew approved these changes Oct 24, 2023

View reviewed changes

tsmaeder and others added 4 commits October 24, 2023 17:15

Initial implementation of Theia test service

f1e5ffe

contributed on behalf of STMicroelectronics Signed-off-by: Thomas Mäder <t.s.maeder@gmail.com>

Testing UI

d85bb1c

Add Testing panel Start the initial tree structure. (same structure as Test API) No label is shown, no active refresh. Basic css implementation

tsmaeder force-pushed the 10669_test_api branch from c8fb1d5 to 008f53f Compare October 24, 2023 15:17

tsmaeder merged commit 439d868 into eclipse-theia:master Oct 24, 2023

vince-fugnitto added this to the 1.43.0 milestone Oct 26, 2023

msujew mentioned this pull request Jan 10, 2024

Use common uuid generator everywhere #13255

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Test API #12935

Implement Test API #12935

tsmaeder commented Sep 24, 2023 •

edited by msujew

Loading

tsmaeder commented Sep 24, 2023

msujew commented Sep 25, 2023

tsmaeder commented Sep 25, 2023

msujew commented Sep 25, 2023

tsmaeder commented Oct 24, 2023

msujew left a comment •

edited

Loading

msujew Oct 24, 2023

msujew Oct 24, 2023

tsmaeder Oct 24, 2023

msujew Oct 24, 2023

tsmaeder Oct 24, 2023

msujew Oct 24, 2023 •

edited

Loading

tsmaeder commented Oct 24, 2023 •

edited

Loading

msujew commented Oct 24, 2023 •

edited

Loading

	label: 'Execute using Profile... '
	label: 'Execute using Profile...'

Implement Test API #12935

Implement Test API #12935

Conversation

tsmaeder commented Sep 24, 2023 • edited by msujew Loading

What it does

How to test

Follow-ups

Review checklist

Reminder for reviewers

tsmaeder commented Sep 24, 2023

msujew commented Sep 25, 2023

tsmaeder commented Sep 25, 2023

msujew commented Sep 25, 2023

tsmaeder commented Oct 24, 2023

msujew left a comment • edited Loading

Choose a reason for hiding this comment

msujew Oct 24, 2023

Choose a reason for hiding this comment

msujew Oct 24, 2023

Choose a reason for hiding this comment

tsmaeder Oct 24, 2023

Choose a reason for hiding this comment

msujew Oct 24, 2023

Choose a reason for hiding this comment

tsmaeder Oct 24, 2023

Choose a reason for hiding this comment

msujew Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

tsmaeder commented Oct 24, 2023 • edited Loading

msujew commented Oct 24, 2023 • edited Loading

tsmaeder commented Sep 24, 2023 •

edited by msujew

Loading

msujew left a comment •

edited

Loading

msujew Oct 24, 2023 •

edited

Loading

tsmaeder commented Oct 24, 2023 •

edited

Loading

msujew commented Oct 24, 2023 •

edited

Loading