File does not exist errors when building with --experimental_merged_skyframe_analysis_execution

[turmanticant]

Description of the bug:

I originally discussed this bug on the bazel slack with the Aspect folks.
https://bazelbuild.slack.com/archives/C04281DTLH0/p1727907042769939

I was experiencing very flaky builds. Over the past month 60% of my builds failed from weird copy errors. Sometimes it was my tsconfig.json like below, other times it didn't say what the file was, but they were all No such file or directory errors.

I have been able to trigger this 100% of the time by clean expunging my cache and rebuilding.

bazel clean --expunge
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
bazel test //... --experimental_merged_skyframe_analysis_execution
Starting local Bazel server and connecting to it...
WARNING: Download from https://index.docker.io/v2/envoyproxy/envoy/manifests/sha256:079085d7f507469704d8f0aedc2b4c191dc38e0f79548c3133f9733fe85f26bf failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Unable to retrieve the manifest. This could be due to authentication problems or an attempt to fetch an image with OCI image media types.

WARNING: The use of Curl fallback is deprecated and is set to be removed in version 2.0. 
For more details, refer to: https://github.com/bazel-contrib/rules_oci/issues/456

WARNING: Download from https://index.docker.io/v2/library/nginx/manifests/sha256:baa881b012a49e3c2cd6ab9d80f9fcd2962a98af8ede947d0ef930a427b28afc failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Unable to retrieve the manifest. This could be due to authentication problems or an attempt to fetch an image with OCI image media types.

WARNING: The use of Curl fallback is deprecated and is set to be removed in version 2.0. 
For more details, refer to: https://github.com/bazel-contrib/rules_oci/issues/456

WARNING: Unable to retrieve the manifest. This could be due to authentication problems or an attempt to fetch an image with OCI image media types.

WARNING: The use of Curl fallback is deprecated and is set to be removed in version 2.0. 
For more details, refer to: https://github.com/bazel-contrib/rules_oci/issues/456

ERROR: /home/<redacted>/Repository/turmantis/collective/BUILD:126:10: Copying file tsconfig.json failed: (Exit 1): coreutils failed: error executing CopyFile command (from target //:tsconfig) 
  (cd /home/<redacted>/.cache/bazel/_bazel_<redacted>/34c6fa7f467484e23b6742c76862b361/execroot/_main && \
  exec env - \
  external/aspect_bazel_lib~~toolchains~coreutils_linux_amd64/coreutils cp tsconfig.json bazel-out/k8-fastbuild/bin/tsconfig.json)
# Configuration: c0d3a92f91c0c2e9b150983da494f2b9f3635c23b46ec79ec60aa3b2e55d7390
# Execution platform: @@platforms//host:host
src/main/tools/process-wrapper-legacy.cc:80: "execvp(external/aspect_bazel_lib~~toolchains~coreutils_linux_amd64/coreutils, ...)": No such file or directory
INFO: Elapsed time: 32.009s, Critical Path: 0.17s
INFO: 31 processes: 28 internal, 3 linux-sandbox.
ERROR: Build did NOT complete successfully
ERROR: No test targets were found, yet testing was requested
FAILED: 
    Fetching repository @@_main~non_module_repos~gcloud_linux_amd64; starting 11s
    Fetching /home/<redacted>/.cache/bazel/_bazel_<redacted>/34c6fa7f467484e23b6742c76862b361/external/_main~non_module_repos~gcloud_linux_amd64; Extracting google-cloud-sdk-490.0.0-linux-x86_64.tar.gz 11s
    Fetching repository @@gazelle~~go_deps~com_google_cloud_go_firestore; starting 9s
    Fetching repository @@rules_go~~go_sdk~collective__download_0; Downloading and extracting Go toolchain 9s
    Fetching repository @@gazelle~~go_deps~com_google_firebase_go_v4; starting 9s
    Fetching repository @@gazelle~~go_deps~com_github_urfave_cli_v2; starting 9s
    Fetching repository @@gazelle~~go_deps~org_golang_x_sync; starting 9s
    Fetching repository @@gazelle~~non_module_deps~bazel_gazelle_go_repository_cache; starting 9s ... (63 fetches)

For the longest time I thought this was related to rules_js/rules_ts. I think it was in bazel 6 I added build --experimental_merged_skyframe_analysis_execution to my rc trying to fight long cc builds. I didn't really think about it after. @fmeum however suggested I try running with --noexperimental_merged_skyframe_analysis_execution.

This immediately resolved my issue.

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Clean building.

bazel clean --expunge
bazel test //... --experimental_merged_skyframe_analysis_execution

I am struggling to find an existing repository I can reproduce this in. I tested out the frontend bazel examples, but failed to reproduce it there. The repo I am getting this in is a good bit more complicated, but I am working to come up with an environment you can poke at.

Which operating system are you running Bazel on?

Fedora 40

What is the output of bazel info release?

release 7.3.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

There have been a few seemingly related issues. As best I can tell these weren't my issue, and I am running versions that these should be fixed in.

aspect-build/rules_js#1412
#22073

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

@joeleba

[haxorz]

--experimental_merged_skyframe_analysis_execution is known as "Skymeld" and is owned by team-Performance

[joeleba]

This is most likely a duplicate of #23743.