The --cache option doesn't work in emcc: emcc.py saves the cache directory into config.CACHE, but tools/cache.py uses its own local CACHE variable

[yurivict]

Please make sure that config.CACHE in emcc.py and CACHE in tools/config.py always refer to the same value.

Version: 3.1.74
FreeBSD 14.1

[sbc100]

Can you explain your use case a little more. tools/cache.py should ideed have its own CACHE variable but its get initialized from one of 4 places in this order:

1. Command line flag (--cache)
2. Environment variable ($EM_CACHE)
3. Config file (CACHE = ..)
4. Default (inside the emscripten directory).

I just confirmed ./emcc --cache=$HOME/foo does work.

I highly recommend just letting emscripten use the default value. Can you explain why the default value doesn't working for your use case?

[yurivict]

In FreeBSD we install emscripten into a read-only location, so the default cache location can't work because it ends up in a read-only location too.

The --cache also doesn't work, and this makes such installation unusable.

[yurivict]

I just confirmed ./emcc --cache=$HOME/foo does work.

Please make the current directory read-only and you will see that --cache really doesn't work.

[sbc100]

I see. That does sounds like reasonable use case.

If you are overriding the cache directory for all users, would it not make sense to add CACHE = to the config file rather than trying to add command line flags?

I wonder why --cache appears to work for me but not in your case. Can you share more? i.e. the full failing command line and config file. How are you specifying the config file?

[yurivict]

I wonder why --cache appears to work for me but not in your case.

Because you use a writable location.

[sbc100]

I wonder why --cache appears to work for me but not in your case.

Because you use a writable location.

I'm must be doing something different to you:

$ cp -ar ./cache/ /tmp/cache
$ rm -rf ./cache/
$ chmod -w -r /tmp/cache
$ ls /tmp/cache/
ls: cannot open directory '/tmp/cache/': Permission denied
$ ./emcc --cache=/tmp/cache ~/test/hello.c
$ echo $?
0

[sbc100]

If I run with -v I can see -L/tmp/cache/sysroot/lib/wasm32-emscripten in the link flags so I know its using the directory I specify with --cache.

(as I said above I think it would be better use the config file in any case).

[yurivict]

You should do something like this:

$ rm -rf cache
$ chmod -R a-w .
$ ./emcc --cache /tmp/cache

This should reproduce the problem.

[sbc100]

This works for me, but I do also need to set EM_FROZEN_CACHE=1 (or FROZEN_CACHE in the config file):

$ chmod -R a-w .
$ EM_FROZEN_CACHE=1 ./emcc --cache=/tmp/cache ~/test/hello.c -o ~/out.wasm
$ echo $?
0

Can you share the full command line along with the output you are seeing? Also, can you share the config file you are using?

[yurivict]

emscripten is installed into /usr/local/lib/emscripten which is read-only.

This command:
$ emcc --cache /home/yuri/.cache/emscriptenx hello_world.c

prints this when the cache directory isn't present:

shared:INFO: (Emscripten: Running sanity checks)
cache:INFO: generating system headers: sysroot_install.stamp... (this will be cached in "/usr/home/yuri/.cache/emscriptenx/sysroot_install.stamp" for subsequent builds)
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/libGL-getprocaddr.a... (this will be cached in "/usr/home/yuri/.cache/emscriptenx/sysroot/lib/wasm32-emscripten/libGL-getprocaddr.a" for subsequent builds)
emcc: error: unable to create cache directory "/usr/local/lib/emscripten/cache": [Errno 13] Permission denied: '/usr/local/lib/emscripten/cache' (see https://emscripten.org/docs/tools_reference/emcc.html for info on setting the cache directory)
emcc: error: unable to create cache directory "/usr/local/lib/emscripten/cache": [Errno 13] Permission denied: '/usr/local/lib/emscripten/cache' (see https://emscripten.org/docs/tools_reference/emcc.html for info on setting the cache directory)
emcc: error: unable to create cache directory "/usr/local/lib/emscripten/cache": [Errno 13] Permission denied: '/usr/local/lib/emscripten/cache' (see https://emscripten.org/docs/tools_reference/emcc.html for info on setting the cache directory)
emcc: error: unable to create cache directory "/usr/local/lib/emscripten/cache": [Errno 13] Permission denied: '/usr/local/lib/emscripten/cache' (see https://emscripten.org/docs/tools_reference/emcc.html for info on setting the cache directory)
emcc: error: subprocess 1/4 failed (returned 1)! (cmdline: /usr/local/lib/emscripten/emcc -c -O2 -Wall -fno-unroll-loops -Oz -fno-inline-functions -g -sSTRICT -Werror -DMAX_WEBGL_VERSION=1 -DGL_ENABLE_GET_PROC_ADDRESS=1 ../../../../../../local/lib/emscripten/system/lib/gl/gl.c)
*** Error code 1

[sbc100]

Ah I see, I think this is because emscripten is calling sub-processes to populate the cache but those don't inherit the --cache command line flag.

Can you trying using EM_CACHE instead? Or using the config file?

[sbc100]

I think it might make sense to remove the --cache flag completely, and instead rely on just the config file or environment variable. This would match the rest of the config.py settings which don't have corresponding command line flags. Would your workflow be able to work without this command line flag?

[yurivict]

Would your workflow be able to work without this command line flag?

Yes.

In fact I've added these 2 relevant patches that put the cache directory into ~/.cache/emscripten by default:

• patch1
• patch2

I think that this is better for most users.

[sbc100]

Awesome, in that case I'll propose removing --cache rather then trying to propagate it.

[sbc100]

Actually I was just looking to this and it looks like we set EM_CACHE so that subprocesses are effected:

emscripten/emcc.py

Lines 1368 to 1373 in ff146f3

	elif check_arg('--cache'):
	config.CACHE = os.path.abspath(consume_arg())
	cache.setup()
	# Ensure child processes share the same cache (e.g. when using emcc to compiler system
	# libraries)
	os.environ['EM_CACHE'] = config.CACHE

. So now I'm confused why it didn't work for... will continue to investigate.

[sbc100]

I'm now trying to reproduce again since I can't see how this could be happening.

$ chmod -R a-w .
$ rm -rf ~/tmp
$ ./emcc ~/test/hello.c --cache=$HOME/tmp -o ~/out.wasm
shared:INFO: (Emscripten: Running sanity checks)
cache:INFO: generating system headers: sysroot_install.stamp... (this will be cached in "/usr/local/google/home/sbc/tmp/sysroot_install.stamp" for subsequent builds)
cache:INFO:  - ok
cache:INFO: generating system asset: sysroot/lib/wasm32-emscripten/crt1.o... (this will be cached in "/usr/local/google/home/sbc/tmp/sysroot/lib/wasm32-emscripten/crt1.o" for subsequent builds)
system_libs:INFO: compiled 1 inputs in 0.23s
cache:INFO:  - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/libGL-getprocaddr.a... (this will be cached in "/usr/local/google/home/sbc/tmp/sysroot/lib/wasm32-emscripten/libGL-getprocaddr.a" for subsequent builds)
system_libs:INFO: compiled 4 inputs in 0.37s
cache:INFO:  - ok
...

Here all the subcommands to build the libraries seem to be using the cache configured with --cache. When you do this somehow the subcommands are not honoring the EM_CACHE in the environment that --cache adds.