EPERM Error messages produced when retyring rename of project-graph.json cause CI pipeline failure

[JonMarbach]

Current Behavior

We have a bit of a complex setup, but in summay we trigger our front-end CI builds via CMake which in turn uses MSBuild on Windows. We recently upgraded to nx v19.4 (from ~v10) and we are now getting intermittent build failures. (Apologies in advance - I'm not super-familiar with nx.)

Before you read further, I'll caution you that this message and stack trace aren't quite what they seem, but the output messages look like this:

nx run appname:build:production

ERROR (1) when writing
EPERM: operation not permitted, rename '.nx\workspace-data\project-graph.json~ea6e942b' -> '.nx\workspace-data\project-graph.json'
CUSTOMBUILD : error : EPERM: operation not permitted, rename '.nx\workspace-data\project-graph.json~ea6e942b' -> '.nx\workspace-data\project-graph.json' [\angular_all_apps.vcxproj]
at renameSync (node:fs:1032:11)
at writeCache (\node_modules\nx\src\project-graph\nx-deps-cache.js:102:39)
at buildProjectGraphAndSourceMapsWithoutDaemon (\node_modules\nx\src\project-graph\project-graph.js:133:44)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async createProjectGraphAndSourceMapsAsync (\node_modules\nx\src\project-graph\project-graph.js:208:25)
at async createProjectGraphAsync (\node_modules\nx\src\project-graph\project-graph.js:197:39)

I've traced the problem to nx-deps-cache.ts ( https://github.com/nrwl/nx/blob/master/packages/nx/src/project-graph/nx-deps-cache.ts ) around line 142:
` try {
writeJsonFile(tmpProjectGraphPath, projectGraph);
renameSync(tmpProjectGraphPath, nxProjectGraph);

  writeJsonFile(tmpFileMapPath, cache);
  renameSync(tmpFileMapPath, nxFileMap);
  done = true;
} catch (err: any) {
  if (err instanceof Error) {
    console.log(
      `ERROR (${retry}) when writing \n${err.message}\n${err.stack}`
    );
  } else {
    console.log(
      `ERROR  (${retry}) unknown error when writing ${nxProjectGraph} and ${nxFileMap}`
    );
  }
  ++retry;
}`

https://github.com/nrwl/nx/blob/71653dcbfe01d29cc60355a7221ab1370d899b29/packages/nx/src/project-graph/nx-deps-cache.ts#L141C5-L159C6

When the code prints err.message to console.log at line 150, the text begins with "Error: " which MSBuild automatically interprets as a failure (I have verified this with a simple test app that prints "Error: This is not an error" - MSBuild flags it as having failed.)

So, while nx is doing the right thing by retrying, it is outputting an error message before it has truly failed, confusing the CI pipeline.

Expected Behavior

Since I'm far from an expert I'm open to suggestions on how to change this. Perhaps it should only output an error message if all the retries have failed. Perhaps there should be some verbosity control or env var which suppresses this message. Perhaps the message should be reformatted to strip "Error:" from the beginning.

At the very least if you could advise how to avoid this, that would be appreciated. For example, I see a recent addition of the environment variable NX_FORCE_REUSE_CACHED_GRAPH by @AgentEnder last month, but I'm not sure what all must be committed (beyond project-graph.json) in order for that to work.

GitHub Repo

No response

Steps to Reproduce

Since this is intermittent and part of a complex proprietary setup with CMake and MSBuild I don't think I can provide a repro case.

Nx Report

Node   : 20.18.0
OS     : win32-x64
npm    : 10.8.2

nx                 : 19.4.4
@nx/js             : 19.4.4
@nx/jest           : 19.4.4
@nx/linter         : 19.4.4
@nx/eslint         : 19.4.4
@nx/workspace      : 19.4.4
@nx/angular        : 19.4.4
@nx/devkit         : 19.4.4
@nx/eslint-plugin  : 19.4.4
@nx/playwright     : 19.4.4
@nrwl/tao          : 19.4.4
@nx/web            : 19.4.4
@nx/webpack        : 19.4.4
typescript         : 5.5.4
---------------------------------------
Registered Plugins:
@nx/playwright/plugin
@nx/eslint/plugin

Failure Logs

Package Manager Version

No response

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

We are hanging back at nx 19.4 due to some bug reports we saw of projects not being recognized. The code in question though has not changed between 19.4 and the latest version.

[JonMarbach]

I misspoke above - MSBuild does not simply look for Error: at the beginning of a line; as per https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-diagnostic-format-for-tasks?view=vs-2022 MSBuild scans build output for messages in the following format:

Origin : Subcategory Category Code : Text

But Subcategory and Code are optional, and Category is Error or Warning (case insensitive). Origin is also optional. So the output message has to be carefully crafted to avoid being parsed as an Error by MSBuild. I tested the following format and it worked well:

nx : warning : Exception thrown when outputting project-graph.json: Error: EPERM: operation not permitted, rename...

MSBuild interpreted it as a warning rather than as an error, which feels appropriate if we want to output something for every retry, not just the final failing retry.