cts is not executed by ci workflow, cts_runner seems broken

[turbocrime]

Description

it seems like the cts workflow was removed, and thus this repository's ci has no way to use cts_runner since 34e947d

34e947d#diff-6867f1034f74cf18cb5e7345b15c2b5ae21eb47bb0a53d2b0c3bb4947753f4b6

cts_runner execution has since broken (rust panic aborts execution instead of simply failing), and before that, some cts pass/fail status regressed.

Expected vs observed behavior

cts_runner should not panic.

ci workflow should at least optionally execute deno cts. ideally, success of each individual test could be recorded periodically, or for each set of changes.

if cts_runner is executed and tracked by some other system, this should be documented and public. if cts_runner is no longer used, it should be removed.

Extra materials

cts_runner panics for everything in test.lst including unittests on 0f5f058 (tip of trunk)

cts_runner passes two sets of tests on 5543961 (firefox dependency)

passing:

unittests:*
webgpu:api,operation,command_buffer,basic:*

panic and abort:

webgpu:api,operation,compute,basic:*

=== Running webgpu:api,operation,compute,basic:* ===
    Finished dev [unoptimized + debuginfo] target(s) in 0.13s
     Running `/Users/yet/Developer/github.com/gfx-rs/wgpu/target/debug/cts_runner ./tools/run_deno --verbose 'webgpu:api,operation,compute,basic:*'`
thread 'main' panicked at /Users/yet/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.272.0/gotham_state.rs:82:3:
required type wgpu_core::global::Global is not present in GothamState container
stack backtrace:
   0: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: deno_core::gotham_state::missing
             at /Users/yet/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.272.0/gotham_state.rs:82:3
   3: deno_core::gotham_state::GothamState::borrow::{{closure}}
             at /Users/yet/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.272.0/gotham_state.rs:42:41
   4: core::option::Option<T>::unwrap_or_else
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/option.rs:976:21
   5: deno_core::gotham_state::GothamState::borrow
             at /Users/yet/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.272.0/gotham_state.rs:42:5
   6: deno_webgpu::compute_pass::op_webgpu_compute_pass_set_pipeline::call
             at /Users/yet/Developer/github.com/gfx-rs/wgpu/deno_webgpu/compute_pass.rs:34:5
   7: deno_webgpu::compute_pass::op_webgpu_compute_pass_set_pipeline::slow_function_impl
             at /Users/yet/Developer/github.com/gfx-rs/wgpu/deno_webgpu/compute_pass.rs:20:1
   8: deno_webgpu::compute_pass::op_webgpu_compute_pass_set_pipeline::v8_fn_ptr
             at /Users/yet/Developer/github.com/gfx-rs/wgpu/deno_webgpu/compute_pass.rs:20:1
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5

exception and failure:

webgpu:api,operation,compute,basic:*
webgpu:api,operation,rendering,basic:clear:*
webgpu:api,operation,rendering,basic:fullscreen_quad:*

webgpu:api,operation,rendering,blending:*
webgpu:api,operation,rendering,blending:GPUBlendComponent:*
webgpu:api,operation,rendering,depth:*
webgpu:api,operation,rendering,draw:*

all seem to fail the same way

=== Running webgpu:api,operation,rendering,basic:clear:* ===
    Finished dev [unoptimized + debuginfo] target(s) in 0.15s
     Running `/Users/yet/Developer/github.com/gfx-rs/wgpu/target/debug/cts_runner ./tools/run_deno --verbose 'webgpu:api,operation,rendering,basic:clear:*'`
[fail] webgpu:api,operation,rendering,basic:clear: (20ms). Log:
  - EXCEPTION: Bad resource ID
    BadResource: Bad resource ID
        at GPURenderPassEncoder.end (ext:deno_webgpu/01_webgpu.js:3833:21)
        at RunCaseSpecific.fn (file:///Users/yet/Developer/github.com/gfx-rs/wgpu/cts/out/webgpu/api/operation/rendering/basic.spec.js:36:8)

** Failures **
[fail] webgpu:api,operation,rendering,basic:clear: (20ms). Log:
  - EXCEPTION: Bad resource ID
    BadResource: Bad resource ID
        at GPURenderPassEncoder.end (ext:deno_webgpu/01_webgpu.js:3833:21)
        at RunCaseSpecific.fn (file:///Users/yet/Developer/github.com/gfx-rs/wgpu/cts/out/webgpu/api/operation/rendering/basic.spec.js:36:8)

** Summary **
Passed  w/o warnings = 0 / 1 =   0.00%
Passed with warnings = 0 / 1 =   0.00%
Skipped              = 0 / 1 =   0.00%
Failed               = 1 / 1 = 100.00%
=== Running webgpu:api,operation,rendering,basic:fullscreen_quad:* ===
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s
     Running `/Users/yet/Developer/github.com/gfx-rs/wgpu/target/debug/cts_runner ./tools/run_deno --verbose 'webgpu:api,operation,rendering,basic:fullscreen_quad:*'`
[fail] webgpu:api,operation,rendering,basic:fullscreen_quad: (24ms). Log:
  - EXCEPTION: serde_v8 error: invalid type; expected: string, got: Number
    TypeError: serde_v8 error: invalid type; expected: string, got: Number
        at GPUDevice.createRenderPipeline (ext:deno_webgpu/01_webgpu.js:1441:26)
        at RunCaseSpecific.fn (file:///Users/yet/Developer/github.com/gfx-rs/wgpu/cts/out/webgpu/api/operation/rendering/basic.spec.js:60:29)

** Failures **
[fail] webgpu:api,operation,rendering,basic:fullscreen_quad: (24ms). Log:
  - EXCEPTION: serde_v8 error: invalid type; expected: string, got: Number
    TypeError: serde_v8 error: invalid type; expected: string, got: Number
        at GPUDevice.createRenderPipeline (ext:deno_webgpu/01_webgpu.js:1441:26)
        at RunCaseSpecific.fn (file:///Users/yet/Developer/github.com/gfx-rs/wgpu/cts/out/webgpu/api/operation/rendering/basic.spec.js:60:29)

** Summary **
Passed  w/o warnings = 0 / 1 =   0.00%
Passed with warnings = 0 / 1 =   0.00%
Skipped              = 0 / 1 =   0.00%
Failed               = 1 / 1 = 100.00%

Platform

i'm running cts on macos sonoma and an m3 pro.

i understand macos was removed from CI test coverage some time ago, but all cargo xtask test (except stack depth) is still successful on 5543961 and tip of trunk.

[turbocrime]

pr to repair cts_runner in #6840 but i will leave ci and workflow to maintainers

[cwfitzgerald]

We haven't really maintained the cts_runner of late, as we have two downstream projects running the cts (firefox and servo) - that said we'd definitely like this to keep working.

[cwfitzgerald]

i understand macos was removed from CI test coverage some time ago

It was added back shortly after, it was an administrative thing

I'll see if I can tackle putting the cts job back