Composing adapters?

[leotaku]

Hello! I've been following the trustfall project for quite some time, and have recently been considering writing some adapters for a set of problems I have. To keep it short, I have a number of databases and APIs that are currently connect using various scripts and ad-hoc programs, and I would like to convert these scripts into trustfall queries. This seems to be exactly what trustfall was made for.

Now, here is my issue: I've been looking at the demo-hytradboi project, and the way it seems to be structured is that one "central" adapter and one vertex type contain all the information about multiple API endpoints. As the number of API endpoints that I want to integrate grows, this setup would seem to become increasingly unmaintainable.

Instead would prefer a project structure where a bunch of distinct trustfall adapters can communicate using a set of "integration points". To take the canoncal HYTRADBOI example, I would imagine a HackerNews adapter that has a Url output vertex and a separate GitHub adapter that can convert Url vertices pointing at GitHub repositories to GithubRepository vertices.

I've been looking at #481 as a possible solution. I would move my "integration vertices" into a separate crate that all adapters can depend on, and then implement AsVertex where necessary. Is this a good idea? Otherwise, are there any examples/best practices on how to compose multiple adapters?

[obi1kenobi]

That is indeed exactly what Trustfall is made for! I'm thrilled you're considering using it for that.

I'd love to hear more about your use case!

• Are you thinking about doing this from Rust, or via one of the other language bindings like Python?
• What data sets are you thinking about targeting?
• Is this something you're building for fun / for personal use, or something bigger?

This sort of composition is exactly what #481 was meant to allow, and I've prototyped some code in that direction — enough to know what you describe is certainly possible. But I'm worried about maintainability, since such a large feature needs to be exhaustively tested and will invariably get a lot of use once it's available — probably even commercial use, based on what I've heard. As a solo maintainer, I'm not in a position to offer free support for commercial companies' use of such a maintenance-intensive feature, so I've been a bit hesitant to publish the work I've done on that until I can figure out a good way to mitigate that risk and the associated risk of burnout for me. If I get burned out from an overwhelming support burden at a "free" price point, it'd be a lose-lose for everyone. This sort of thing has happened to other projects in the past, so it's not a theoretical risk and it would be irresponsible of me to not take it seriously.

[leotaku]

Thank you for the quick and detailed reply! To answer your questions:

• My plan is to use Rust directly.
• I plan to query mostly personal data sources, such as my mailbox, calendar, local structured text files, posts on social networks, etc.
• This is just something I'm building for fun right now.

I can definitely empathize with your concerns about maintainability and burnout. While I can't say I've ever tried to make any of my open source projects a sustainable source of income, I really respect you for trying to achieve that with trustfall! As for maintainability concerns related to this feature specifically, I suppose I can't really comment as I'm of course not familiar with your unpublished work. 😅

If you allow me to guess, I suppose your concerns about releasing/endorsing such a feature lie within the increased flexibility given to code/adapters written by other people leading to increased maintenance burden on you, more so than the inherent complexity of the feature itself?

[obi1kenobi]

Both are of concern.

To give an example of inherent complexity causing concern, consider what happens once there are modular adapters out there:

• What if two adapters' schemas both define the same type, how would one compose them? The answer is "rename one of the types in one of the adapters" but that's a feature that has to be designed, implemented, and tested.
• What if one adapter wants to extend the schema of another, e.g. by providing additional properties or edges? That's again something that has to be designed, implemented, and tested.
• What if multiple adapters want to use a common interface type like Webpage, and implement that interface on their own types? How do we make sure their idea of Webpage is the same for all adapters? Again: thought through, designed, implemented, tested.
• What about performance? Right now we have an optimizations API, but can that work across adapters too? Can adapters "observe" that they are being composed with another source of data? How do we walk the fine line between exposing as much info as possible for the benefit of optimizations, vs making additional maintainability and semver hazards that may make composition brittle?

These are all features I sorely, desperately want to exist. I'd use them in my own code and in my own tools if they existed. But I've just described several million USD worth of development and maintenance work, when I'm in a situation where my income doesn't come close to even paying my rent.

At the moment, Trustfall is in a relatively happy spot: it works very well, it has no known query execution bugs, it's fast. It could have much better docs and a better story for being integrated into other languages, like Python or JS or WASM. But aside from that bit of adoption friction, I don't think anyone would have a strongly negative experience with Trustfall right now.

Officially supporting composition will 50x the scope of the project, and I don't want to make that jump until I'm reasonably confident it won't upset that balance. I'd be really sad (and would quickly get burnt out) if people coming to the Trustfall project all of a sudden started having negative experiences with it left, right, and center. I really dislike shipping half-baked things!

I'm hoping to get the project into a more self-sustainable spot before "officially" supporting composition, both monetarily and onboarding/support-wise. For example, if we had more and better docs, or clearer error messages, that would ease the burden on me and free me up to build other things. If that's something you're interested in, I'd love your help! If not, that's completely alright too.

In the meantime, I'd recommend the "one adapter that knows how to talk to all the data sources" approach as the most sustainable and least prone to issues.