[Bazel CI] aspect_rules_js no such file error or directory with Bazel@HEAD in CI

[sgowroji]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3711#018e2bad-eced-489f-aa96-d3974826f324

Platform: MacOS

Logs:

ERROR: �[0m/Users/buildkite/builds/bk-imacpro-17/bazel-org-repo-root/examples/frontend/BUILD.bazel:10:22: Copying directory aspect_rules_js~~npm~npm__typescript__4.9.3/package [for tool] failed: Exec failed due to IOException: /private/var/tmp/_bazel_buildkite/234bc2f97ee57c6cd035cc0acf2083ba/execroot/_main/external/aspect_rules_js~~npm~npm__typescript__4.9.3/package (No such file or directory)
ERROR: �[0m/Users/buildkite/builds/bk-imacpro-17/bazel-org-repo-root/examples/frontend/BUILD.bazel:10:22: Copying directory aspect_rules_js~~npm~npm__he__1.2.0/package [for tool] failed: Exec failed due to IOException: /private/var/tmp/_bazel_buildkite/234bc2f97ee57c6cd035cc0acf2083ba/execroot/_main/external/aspect_rules_js~~npm~npm__he__1.2.0/package (No such file or directory)
ERROR: �[0m/Users/buildkite/builds/bk-imacpro-17/bazel-org-repo-root/examples/frontend/vue/libraries/simple/BUILD.bazel:55:12 Copying files to directory vue/libraries/simple/simple failed: Exec failed due to IOException: /private/var/tmp/_bazel_buildkite/234bc2f97ee57c6cd035cc0acf2083ba/execroot/_main/external/aspect_rules_js~~npm~npm__typescript__4.9.3/package (No such file or directory)
ERROR: �Build did NOT complete successfully

Culprit:

Steps:

git clone https://github.com/bazelbuild/examples
git reset 7285d2e400b04f81df4d9edcad5a8fc8d43c607c  --hard
export USE_BAZEL_VERSION=3697397569e25a29a828fb46b26302876d372e26
bazel build ... 

CC Greenteam @Wyverald

[Wyverald]

The same failure happens on release Bazel (https://buildkite.com/bazel/bazel-bazel-examples/builds/2766#018e2acf-d2b3-4ad7-88ae-e73c3f46c796), so I think we should just move this issue to the Bazel examples repo? also cc @alexeagle

[Wyverald]

sorry, I didn't realize we were already in the examples repo...

[meteorcloudy]

sorry, I didn't realize we were already in the examples repo...

Np! ;D

@alexeagle Any idea on how to address this flaky failure?

[alexeagle]

@gregmagolan and @tjgq have had an email thread about it. I think it's a matter of finding some time for it. Our clients observe it sometimes too, we are just having trouble understanding the cause.

[sgowroji]

Hi Team, Any Update/progress on the above issue ?

[gregmagolan]

1.40.0 rules_js release has a fix/work-around for the issue for most cases. See the PR summary of aspect-build/rules_js#1538 for more info.

[fmeum]

I'm pretty sure this is caused by a race in skymeld that isn't related to caching and can't be worked around reliably: bazelbuild/bazel#22073

[fmeum]

@alexeagle Did you mean to close this with #445?

[alexeagle]

Yes, that change includes the version bump for rules_js that @gregmagolan mentioned 4d ago

[fmeum]

Yes, that change includes the version bump for rules_js that @gregmagolan mentioned 4d ago

Based on bazelbuild/bazel#22073, I don't think that this can be worked around reliably on the ruleset side, it appears to be an honest race in Skymeld that can affect virtually any external repo. I would suggest keeping this open until it doesn't reproduce in CI anymore.

[alexeagle]

Okay sure, but does that "honest race in Skymeld" have an owner? I don't think Greg or I are planning to do more here, and I don't love having an issue "rules_js is broken" on the examples repo.

[fmeum]

As far as I know @joeleba is actively working on bazelbuild/bazel#22073.

[sgowroji]

Examples CI is Green now. We can close this issue. Thank you all for the great support.