I'm interested more in how to use whatwg/fs with sync access handle

[jimmywarting]

I plan to run sqlite in a web worker and i also want to use the new whatwg/fs.

I know that whatwg/fs has support for reading/writing files synchronous if you call: fileHandle.createSyncAccessHandle()
and it also allows for things such as "write in place" and manually flushing data or what not.

I was wondering: can i use wa-sqlite in a web worker and using a sync access handle as my preferred database without having some penelty from the asyncify? i still only want the sync part of it all

[rhashimoto]

[Edit: This initial reply is mostly wrong.]

I think createSyncAccessHandle() has only been implemented in Chromium-based browsers in an origin trial, but yes, if that restriction doesn't bother you then you should be able to write a VFS that works without Asyncify. Note that the origin trial ends in October.

You might want to look at the example VFS that uses asynchronous access handles as a starting point.

[rhashimoto]

The added overhead from Asyncify is often not as bad as people expect. Most database applications are limited by storage performance and not CPU performance. The File System API is pretty slow (slower than IndexedDB can be, actually), even with sync access handle, so the incremental time that Asyncify adds is relatively small.

[quolpr]

Yep, I totally agree with @rhashimoto, the overhead is not so big

[jimmywarting]

Good to know

[jimmywarting]

The File System API is pretty slow (slower than IndexedDB can be, actually), even with sync access handle

That's pretty bad, how the heck can they design a file system that is slower than using a virtual FS with indexeddb?! can't understand it. i get it that they want to design a secure file system and scanning files and doing atomic writes... it can hurt perfomences so badly depending on what you want to do.

i hope the new suggestions will improve stuff. have long been waiting for the "in-place" option for some other parts of my application

[rhashimoto]

The fact that my OPFS VFS was slower than my IndexedDB VFS was initially surprising to me, too, but actually it makes sense. First of all, the OPFS implementations are somewhat new while IndexedDB has been around since 2015 or so, so hopefully we'll see optimizations over time. But also remember that I'm writing filesystems specifically for SQLite, so it's possible that the File System Access API could be faster for other usages.

One reason my OPFS VFS is slower is because I use a non-synchronous access handle for database file reads. The reason I don't use a synchronous access handle instead is to allow multiple connections to share a read lock for concurrent querying.

Another reason my OPFS VFS is slower is there are no write guarantees until a successful flush. This means that any interruption can leave the database file in an inconsistent state so you also need a persistent journal file. The ability to make an atomic write to IndexedDB means that there are several implementation strategies that remove the need to write (and potentially read) a persistent journal file for many/most cases without sacrificing consistency. Using batch atomic writes is one approach.

Some IndexedDB implementations allow trading durability for performance, but synchronous access handles do not. Timing comparisons using IndexedDB relaxed durability aren't really apples-to-apples but it is another advantage for IndexedDB if durability isn't a requirement.

You can compare the performance of OPFS here (on Safari and Chromium-based browsers) with IndexedDB here.

[rhashimoto]

@jimmywarting This discussion (okay, more of a soliloquy) on changes to the OPFS API may be of interest to you.

[jimmywarting]

Cool, Thanks for sharing.