H-3383: Implement harpc service trait

[indietyp]

🌟 What is the purpose of this PR?

This implements the harpc service trait. This is the machinery a user (or proc-macro) is required to implement. While this may not look like much, it actually took quite a bit longer to implement, due to considerations of how to do encoding of payload and having a working example.

The example examples/account.rs shows what an example would look like.

This service trait machinery can be split into separate primary parts:

• user defined trait
  • instead of following a function based design, like axum, we use a tarpc like approach, where the user defines a trait that he then implements. This trait is split into two distinct roles with the same signature. Server and Client the code for the client can be easily generated (if wanted) and only the server side needs to be implemented by the user.
• ServiceDelegate, the service delegate is there to provide a bridge between the typed trait interface and the tower service. it takes just a handful of arguments and then depending on the required argument decodes them. As shown in the example implementing this per hand is quite straight forward, but it would also be possible to do this via machine generated code quite easily, although it would most likely require some sort of procedural macro.
• Service and Procedure trait. Those are not strictly needed, but allow us to, at a later stage, verify some additional information, such as: the current version vs the requested version, procedures that are part of it, and most importantly it allows us to have a bidirectional type level link between the payload (procedure) and every service, through the use of marker traits. The Procedure trait and implementation are not strictly needed, but included here as they allow us to provide better documentation for the user (e.g. when was this method added, etc.)

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

• are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 20.18%. Comparing base (f3d984a) to head (6551827).
Report is 49 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5283      +/-   ##
==========================================
+ Coverage   19.66%   20.18%   +0.51%     
==========================================
  Files         510      510              
  Lines       17156    17300     +144     
  Branches     2546     2546              
==========================================
+ Hits         3374     3492     +118     
- Misses      13761    13770       +9     
- Partials       21       38      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[TimDiekmann]

Looks good so far, very exciting that we see a service now!

A few comments mainly to clarify a few things. I'm not sure how useful the metadata will be initially because we won't have a versioned API, yet.

[TimDiekmann]

Can we add a documentation for this trait please?

[indietyp]

see: 350295c

[TimDiekmann]

Why we cannot use tuples here?

[indietyp]

We could, but I found that a HList (or similar h-list implementation) is just a lot easier to work with. With tuples you have some unique problems/constraints, like:

• to implement them you need to use a macro, the code is very repetitive
• you have an upper limit (like 16). You can circumvent that limit by using more traits / a proxy trait, but that is just very messy in general and hard to decipher.

That's why I semi recently-ish pivoted to use HList from frunk for these kind of things, instead of using tuples. It's a lot easier to write and reason about and the HList! and hlist! macro make it easy to construct these types. (also the crate itself has some cool things like coproduct, but that's beside the point).

I found the code to just be a lot more readable on the implementation side, with a bit of a negative (because now you're compelled to use a type-level macro) but found that the tradeoffs were good enough, if you disagree please let me know and I'll try to use tuples here.

[TimDiekmann]

We can stick with HList for now so we can make progress here, but generally, I prefer language-constructs (such as tuples). I don't see why construction of HList is easier when in a constant scenario such as defining Procedure. In addition, the documentation is much easier to read when looking at tuples and if we use a macro to implement a trait on each tuple elements that's fine. Something along these lines:

// Potential user-defined implementation if required
impl Trait for CreateAccount {...}

impl<T: Trait> Trait for (T,) { ... }
impl<T: Trait, U: Trait> Trait for (T,U) { ... }

while the Trait bound might not be needed.

Let's keep it or now, but let's also consider changing it later. Generally, we don't need to create procedures dynamically I guess? If we build it dynamically we could use frunk as a helper to construct a tuple.

[indietyp]

Yes. The reason I like this approach more is that it allows for a clean separation of concerns.

It allows us to have two different traits, one for just the list, another one for the trait itself and it doesn't require a macro to implement everything.

I also found that during dynamically building up a list it's a bit nicer.

I also fully get your point! In a perfect would we would have variable sized tuples we could implement traits over.

[TimDiekmann]

I also fully get your point! In a perfect would we would have variable sized tuples we could implement traits over.

Variadic generics or at least variadic tuples would be a game changer for many things 🚀

[TimDiekmann]

I don't really understand this. How I can get information for a removed procedure?

[indietyp]

It's about when something is scheduled for removal, like on v6.0 this will be removed (a future version).

[TimDiekmann]

I don't think this is needed. A deprecation warning should be good enough for that. If a function is stable in v3, then it's deprecated in v4, then it's expected to be removed later.

[indietyp]

you are right, done in 6551827

[TimDiekmann]

Can we add names to variables here?

[indietyp]

see: d3cdbb9

[TimDiekmann]

Happy to keep Removal in, but also happy if it would go away 😉
Let's defer HList->tuple (discussion and potential switch) to a later point when we have more services.

[github-actions]

Benchmark results

@rust/graph-benches – Integrations

representative_read_multiple_entities

Function	Value	Mean	Flame graphs
entity_by_property	depths: DT=0, PT=2, ET=2, E=2	$$54.4 \mathrm{ms} \pm 268 \mathrm{μs}\left({\color{gray}0.493 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=0, PT=0, ET=2, E=2	$$50.4 \mathrm{ms} \pm 348 \mathrm{μs}\left({\color{gray}0.801 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=0, PT=0, ET=0, E=2	$$43.5 \mathrm{ms} \pm 177 \mathrm{μs}\left({\color{gray}-0.272 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=255, PT=255, ET=255, E=255	$$67.3 \mathrm{ms} \pm 373 \mathrm{μs}\left({\color{gray}-0.517 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=2, PT=2, ET=2, E=2	$$59.2 \mathrm{ms} \pm 395 \mathrm{μs}\left({\color{gray}-0.341 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=0, PT=0, ET=0, E=0	$$39.5 \mathrm{ms} \pm 126 \mathrm{μs}\left({\color{gray}-0.201 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=0, PT=2, ET=2, E=2	$$94.0 \mathrm{ms} \pm 369 \mathrm{μs}\left({\color{gray}-1.076 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=0, PT=0, ET=2, E=2	$$90.5 \mathrm{ms} \pm 367 \mathrm{μs}\left({\color{gray}0.516 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=0, PT=0, ET=0, E=2	$$79.3 \mathrm{ms} \pm 210 \mathrm{μs}\left({\color{gray}-0.709 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=255, PT=255, ET=255, E=255	$$107 \mathrm{ms} \pm 587 \mathrm{μs}\left({\color{gray}0.976 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=2, PT=2, ET=2, E=2	$$99.4 \mathrm{ms} \pm 635 \mathrm{μs}\left({\color{gray}0.496 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=0, PT=0, ET=0, E=0	$$42.3 \mathrm{ms} \pm 200 \mathrm{μs}\left({\color{gray}0.061 \mathrm{\%}}\right) $$	Flame Graph

representative_read_entity

Function	Value	Mean	Flame graphs
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1	$$15.9 \mathrm{ms} \pm 149 \mathrm{μs}\left({\color{gray}-2.473 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1	$$15.7 \mathrm{ms} \pm 186 \mathrm{μs}\left({\color{gray}4.69 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1	$$15.7 \mathrm{ms} \pm 184 \mathrm{μs}\left({\color{red}17.2 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1	$$15.9 \mathrm{ms} \pm 168 \mathrm{μs}\left({\color{lightgreen}-5.315 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1	$$15.6 \mathrm{ms} \pm 190 \mathrm{μs}\left({\color{lightgreen}-24.075 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2	$$16.0 \mathrm{ms} \pm 167 \mathrm{μs}\left({\color{gray}0.904 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1	$$16.5 \mathrm{ms} \pm 177 \mathrm{μs}\left({\color{lightgreen}-13.285 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1	$$15.8 \mathrm{ms} \pm 159 \mathrm{μs}\left({\color{lightgreen}-5.273 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1	$$15.2 \mathrm{ms} \pm 158 \mathrm{μs}\left({\color{gray}-4.586 \mathrm{\%}}\right) $$	Flame Graph

representative_read_entity_type

Function	Value	Mean	Flame graphs
get_entity_type_by_id	Account ID: d4e16033-c281-4cde-aa35-9085bf2e7579	$$1.41 \mathrm{ms} \pm 4.08 \mathrm{μs}\left({\color{gray}-0.385 \mathrm{\%}}\right) $$	Flame Graph

scaling_read_entity_complete_zero_depth

Function	Value	Mean	Flame graphs
entity_by_id	10 entities	$$2.07 \mathrm{ms} \pm 9.84 \mathrm{μs}\left({\color{gray}0.231 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	25 entities	$$2.71 \mathrm{ms} \pm 71.5 \mathrm{μs}\left({\color{lightgreen}-10.773 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	5 entities	$$1.90 \mathrm{ms} \pm 9.83 \mathrm{μs}\left({\color{gray}-0.677 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	50 entities	$$3.98 \mathrm{ms} \pm 24.0 \mathrm{μs}\left({\color{gray}-3.604 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	1 entities	$$1.88 \mathrm{ms} \pm 6.89 \mathrm{μs}\left({\color{gray}-0.158 \mathrm{\%}}\right) $$	Flame Graph

scaling_read_entity_complete_one_depth

Function	Value	Mean	Flame graphs
entity_by_id	10 entities	$$51.3 \mathrm{ms} \pm 154 \mathrm{μs}\left({\color{gray}-1.624 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	25 entities	$$73.3 \mathrm{ms} \pm 340 \mathrm{μs}\left({\color{gray}-0.545 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	5 entities	$$24.9 \mathrm{ms} \pm 127 \mathrm{μs}\left({\color{gray}-0.415 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	50 entities	$$268 \mathrm{ms} \pm 1.74 \mathrm{ms}\left({\color{gray}-1.375 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	1 entities	$$19.9 \mathrm{ms} \pm 86.6 \mathrm{μs}\left({\color{gray}-1.145 \mathrm{\%}}\right) $$	Flame Graph

scaling_read_entity_linkless

Function	Value	Mean	Flame graphs
entity_by_id	10000 entities	$$12.5 \mathrm{ms} \pm 125 \mathrm{μs}\left({\color{gray}0.841 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	10 entities	$$1.86 \mathrm{ms} \pm 5.32 \mathrm{μs}\left({\color{gray}-0.546 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	100 entities	$$2.03 \mathrm{ms} \pm 9.44 \mathrm{μs}\left({\color{gray}0.690 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	1000 entities	$$2.77 \mathrm{ms} \pm 13.7 \mathrm{μs}\left({\color{gray}0.740 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	1 entities	$$1.86 \mathrm{ms} \pm 5.94 \mathrm{μs}\left({\color{gray}-0.190 \mathrm{\%}}\right) $$	Flame Graph