Issues with CPU usage in iOS & Safari. Occasional crash.

[tantaman]

I think this is a Safari JIT issue rather than wa-sqlite but wanted to make you aware and see if you have thoughts on how we could move forward.

---

Background

Myself and others started noticing that after doing about a hundred inserts + selects, Safari starts hammering the CPU at ~300-400%

These are all inside of "WebKit" threads

Usage continues to stay high for ~30 seconds after all interactions with the database are complete. After that 30 seconds, usage begins to settle down.

Re-running the same inserts & selects again after the CPU has settled, usage is more reasonable and stays < %150. WebKit threads at this time are doing almost no work.

You can repro this by commenting out all benchmark cases except IDBBatchAtomic and running through that. I'd expect ~100% CPU while running the benchmark but not the 300, 400, sometimes almost 500%. That and I wouldn't expect usage to stay elevated after the benchmark is complete.

You can repro with any case that does a large number of inserts and selects, doesn't need to be the benchmarks and these db events can be spaced out over tens of seconds. They don't have to be going as fast as possible. I.e., cases where Firefox or Chrome never go above 10% CPU, Safari will quickly climb to 300-400%. Safari will also sometimes crash during this.

Chrome & Firefox don't have these issues.

Investigation

My theory is that the problem is caused by the safari JIT given:

1. Problems do not start until running a code path many times
2. User interactivity with the page, and responses from the DB, are not interrupted so this doesn't seem like a problem in the WASM file itself nor any of the JavaScript wrapping it.
3. CPU usage is maxing out more than 1 core. The JS and WASM are all run in a single thread so they would not be able to cause this much load.
4. The crash reports I've gotten during periods of elevated CPU are all in WASM helper threads running through BBQ and OMG paths: https://webkit.org/blog/7691/webassembly/
5. The CPU usage remains elevated even when all scripts are idle
6. Re-running the exact same code after CPU usage has settled down produces CPU usage more in line with Firefox and Chrome

The elevated CPU wouldn't be so problematic except on mobile devices where the "30 seconds" can be upwards of minutes and the fact that Safari sometimes crashes during this.

As I was writing this, one thought did come to mind. Maybe this issue doesn't happen with the non-asyncify build (or is ameliorated to some extent) given the WASM is much smaller and presumably the code is more straightforward to optimize.

[tantaman]

Here's the stack from the crashed thread:

Thread 10 Crashed:: Wasm Worklist Helper Thread
0   JavaScriptCore                	       0x1cb6fb9b8 unsigned short* WTF::Vector<unsigned short, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>::expandCapacity<(WTF::FailureAction)0>(unsigned long, unsigned short*) + 340
1   JavaScriptCore                	       0x1cb6e380c void JSC::B3::Air::(anonymous namespace)::GraphColoringRegisterAllocation::allocateOnBank<(JSC::B3::Bank)0>() + 36512
2   JavaScriptCore                	       0x1cb6e380c void JSC::B3::Air::(anonymous namespace)::GraphColoringRegisterAllocation::allocateOnBank<(JSC::B3::Bank)0>() + 36512
3   JavaScriptCore                	       0x1cb6cd7f8 JSC::B3::Air::allocateRegistersByGraphColoring(JSC::B3::Air::Code&) + 2192
4   JavaScriptCore                	       0x1cb73a8f0 JSC::B3::Air::prepareForGeneration(JSC::B3::Air::Code&) + 3008
5   JavaScriptCore                	       0x1cb635e28 JSC::B3::prepareForGeneration(JSC::B3::Procedure&) + 88
6   JavaScriptCore                	       0x1cc1a3a0c JSC::Wasm::parseAndCompileB3(JSC::Wasm::CompilationContext&, JSC::Wasm::FunctionData const&, JSC::Wasm::TypeDefinition const&, WTF::Vector<JSC::Wasm::UnlinkedWasmToWasmCall, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&, JSC::Wasm::ModuleInformation const&, JSC::Wasm::MemoryMode, JSC::Wasm::CompilationMode, unsigned int, std::__1::optional<bool>, unsigned int, JSC::Wasm::TierUpCount*) + 948
7   JavaScriptCore                	       0x1cc2aae14 JSC::Wasm::OMGPlan::work(JSC::Wasm::Plan::CompilationEffort) + 180
8   JavaScriptCore                	       0x1cc2d8efc JSC::Wasm::Worklist::Thread::work() + 132
9   JavaScriptCore                	       0x1cb1bb3e4 WTF::Detail::CallableWrapper<WTF::AutomaticThread::start(WTF::AbstractLocker const&)::$_0, void>::call() + 480
10  JavaScriptCore                	       0x1cb211f18 WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*) + 244
11  JavaScriptCore                	       0x1cb0587bc WTF::wtfThreadEntryPoint(void*) + 16
12  libsystem_pthread.dylib       	       0x1b1f3d06c _pthread_start + 148
13  libsystem_pthread.dylib       	       0x1b1f37e2c thread_start + 8

[rhashimoto]

Well, that's annoying. Just to be clear, you see this only with Safari on iOS? Safari on Mac and Firefox/Chrome on iOS are fine? I don't have an iOS device so I have very little experience with them.

Your analysis looks plausible to me, except that unless there has been an Apple policy change, third-party browsers on iOS have to use Apple's browser engine instead of their own. I would think Chrome and Firefox on IOS should show the same behavior. Maybe Apple doesn't give third-party browsers access to the same optimizations?

I did at one time see long WASM loading delays and crashes using Chrome on Android devices. This might have been an ARM code generation issue because I couldn't make it happen on the Android emulator. It went away in later versions without being fully investigated, but it sounded like register allocation was a contributor there as your crash stack also shows. It seems like SQLite WASM can put a lot of stress on code generators.

Have you submitted a WebKit bug? If so, then please link it here. WebKit development is often frustratingly opaque but they have been quite responsive to recent bug reports I submitted for which I provided test pages that demonstrated a problem.

Off the cuff ideas:

• Try using IDBMinimalVFS instead of IDBBatchAtomicVFS. If the problem doesn't appear with IDBMinimalVFS, that might bring IDBBatchAtomicVFS or its IndexedDB load under suspicion because the WASM operations should be the same.
• Try building wa-sqlite without optimization or with different Emscripten versions. That might provide a temporary workaround, and maybe hint where the problem is to the WebKit folks.

[tantaman]

Just to be clear, you see this only with Safari on iOS? Safari on Mac and Firefox/Chrome on iOS are fine? I don't have an iOS device so I have very little experience with them.

Everything (Chrome, Firefox, Safari) is broken on iOS due to the fact that they're all just webkit. Only Safari is broken on OS X.

Have you submitted a WebKit bug?

Not yet but I'll do that shortly.

Example of the issue on Safari on OS X:

output.webm

first run hits 500% CPU, 400% of the CPU being taken by WebKit Threads. Next run is ~140% total CPU, WebKit threads taking 4%.

[rhashimoto]

I'm not seeing it on my 2014 Mac mini. After it happens, how do you "reset" Safari so it happens again? Is reloading the page or restarting the browser sufficient (I tried both), or do you need to remove all the cached data?

I'm wondering if it might be architecture-specific like the Chrome problem appeared to be. I'm running Safari 16.5 on a 3 GHz Dual-Core Intel Core i7. Do you see the behavior on both Intel and Apple Silicon Macs?

It's also possible that Safari thinks my Mac is too old and slow for code regeneration to be worthwhile. It does have plenty of memory so I don't think that's why it would give up.

[tantaman]

I'm wondering if it might be architecture-specific like the Chrome problem appeared to be. I'm running Safari 16.5 on a 3 GHz Dual-Core Intel Core i7. Do you see the behavior on both Intel and Apple Silicon Macs?

I've only tested on my m1. Given you don't see it on your mac mini, and the jit would be significantly different between architectures, seems reasonable.

Is reloading the page or restarting the browser sufficient

Yep, reloading is sufficient. Which is unfortunate. If Safari cached this work it'd be a bit less problematic.

[rhashimoto]

It looks like there is supposed to be a way to disable either or both of the BBQ and OMG JIT compilers in Safari on Mac, but I have not yet succeeded in doing that. I'm just documenting what I tried here for anyone who has better ideas to make it work.

WebKit JavaScript has a whole bunch of runtime options that can apparently be overridden by setting an environment variable.

Here are the ones that look interesting (they are both true by default):

JSC_useBBQJIT - allows the BBQ JIT to be used if true
JSC_useOMGJIT - allows the OMG JIT to be used if true
...

The remaining problem is how to start Safari with custom environment variable settings like these. This is the part I seem to be failing at. There seem to be a number of ways to set environment variables for a Mac GUI-based app that have come and gone over the years. It seems like launchctl setenv VAR VALUE and then launch with Spotlight is the latest method. This does require disabling System Integrity Protection.

Unfortunately, I couldn't make this work. I tried sudo launchctl setenv JSC_useWebAssembly false (after disabling SIP) and opening Safari via Spotlight. I expected this setting to disable WASM completely, but it didn't.

My next theory was that the production version of Safari didn't access the external environment, so I downloaded a WebKit nightly build but I couldn't get that to run at all. After bypassing all the unsigned framework loading hurdles, the dynamic loader failed because of a missing symbol when looking in a framework that wasn't included in the nightly bundle:

dyld[20792]: Symbol not found: (__WKWebsiteDataTypeFileSystem)
  Referenced from: '/Library/Apple/System/Library/StagedFrameworks/Safari/Safari.framework/Versions/A/Safari'
  Expected in: '/Users/roy/Downloads/WebKit265616@main/Release/WebKit.framework/Versions/A/WebKit'

I'm guessing that error is related to the fact that recently Safari Technology Preview began telling me that it won't update on macOS Monterey, the last version supported on my hardware (yes, it's probably time for new hardware). Anyway, that's where I'm stuck for now.

If that could be made to work, someone capable of reproducing the original bug could check if disabling OMG made it go away. If yes, that would be strong support for the theory that the bug is caused by runaway recompilation. I know there's already a fair bit of evidence for that, but sometimes people need extra convincing when you have an isolated case they haven't seen before.