Observing changes to data in a quadstore database?

[micahscopes]

Is there some way to observe all mutations on a given scope, graph or query?

I need to synchronize between quadstore and other representations of the data used elsewhere in the application. I'm hoping there might be a way to use quadstore as source of truth if this is possible.

[gsvarovsky]

I know someone working on that 🙂 @Peeja

Spec: https://github.com/m-ld/sustainable-web-apps/blob/main/spec/design/xql.md
Implementation: https://github.com/m-ld/xql

any application will need:

• To be able to query the graph to get exactly the data they need, and
• To get updates to that data as soon as they become available, in a form that makes them easy to present to the user.

The repo is in the m-ld org but the query language (soon to be called QLD) is intended to work on other stores, including raw quadstore.

[jacoscaz]

Hi @micahscopes! At the moment quadstore does not support native replication. I've played around with the idea multiple times in the past, though.

I've had some luck wrapping quadstore's API into an emitter of RDF Patch - https://afs.github.io/rdf-delta/rdf-patch.html - over WebSocket for simpler projects. However, @gsvarovsky and the m-ld crew are lightyears ahead of anything I've ever done in this space.

[micahscopes]

@gsvarovsky very cool! Will this xql/qld thing also stream events for deletions and mutations? By the way, I was just taking a look at m-ld before you messaged... So fun that you of all people responded to my question 😁

@jacoscaz thank you! I was looking at graphs.js earlier and realizing that the diffing/set-like operations could also be really useful for what I'm doing. This usage rdf-patch idea of yours reminds me of that, I'm glad to hear you had some luck with this approach. Do you happen to have a link to your wrapper code that I might find inspiring?

[jacoscaz]

Do you happen to have a link to your wrapper code that I might find inspiring?

@micahscopes unfortunately not, it's not public code. It was really simple, though. The method being wrapped would determine whether quads were being added or removed and the wrapper would first call the actual method on the Quadstore instance and then serialize the quads as RDF Patch and send the results through the socket. There was little to nothing in terms of resiliency against packet loss and things like that but it was enough for simple reactive behavior (i.e. turn on light if presence is detected in the room).

[gsvarovsky]

Will this xql/qld thing also stream events for deletions and mutations?

Sort of: QLD establishes a snapshot of the data that matches the query, and then notifies changes as the next snapshot (as efficiently as possible). So, mutations become changed properties, and deletions become missing data.

This is the model we've found best serves reactive user interfaces. In contrast, m-ld's native core notifies updates as SPARQL Update-like delete/insert patches. But obviously the core is only available if you're using m-ld! Something similar could be shimmed over native quadstore, and that sounds a lot like what @jacoscaz was doing.

By the way, I was just taking a look at m-ld before you messaged

👋🙂

[micahscopes]

Interesting... When you say "snapshot" what exactly are you referring to? A pointer to some representations in a datastore? Or a text serialization? Similarly, do query observables emit serialized updates or pointers to the store?

I've been thinking lately about the potential to build something like this on top of prolly trees, and though I don't know enough yet to say if this is a good approach it sounds promising from what I've heard. The gist is that prolly trees let you efficiently diff, store and gossip about lots of mutations of whatever data structure you implement on top of them.

Sounds like at some point maybe I could use QLD on top of quadstore, graphy.js or even some hypothetical prolly tree based graph store?