IPIP-330: ordered car responses draft #330
Conversation
Jorropo
changed the title
Jorropo
changed the title
|
Having "include identity CIDs" would be a helpful permutation to add to the list here. Ideally we could use this to describe the Filecoin online deal CAR creation for the purpose of generating an agreed CommP (essentially a hash of the CAR on both client and server). Aside from DAG ordering problems, the other problem we've run in to is where one side includes identity CIDs and the other doesn't. For historical reasons, inclusion of identity CIDs got baked in to the online deal flow. But, perhaps with a mechanism to describe what an "ordered CAR" looks like for setting up a Filecoin deal we could have a way of agreeing up-front on whether it should include them or not (previously I've thought about having some kind of boolean flag to set up agreement, but the descriptors here might be useful for this too). |
|
@rvagg I meant (but forgot) to precise that inline CIDs MUST NOT be transmited (but MUST be traversed). I was thinking suffixes could actually be made a list of flags if needed, sounds like legacy Filecoin code would need it so a Then Filecoin deals could be either one of:
(other ones aren't acceptable because they are not fully deterministic) Does that sounds good ? |
Jorropo
force-pushed
the
ordered-car-responses
branch
from
499f211 to
8a525a4
Compare
Jorropo
force-pushed
the
ordered-car-responses
branch
from
8a525a4 to
57d44e6
Compare
|
Also if this would be usefull for filecoin, the ordering part of this spec should be moved into IPLD. |
|
@Jorropo filecoin is stedily moving away from considering the ordering of blocks within a .car file and/or whether they form an actual DAG. What you are discussing here is strictly of interest to gateways and out of scope for bulk transport systems ( like Fil ). |
|
@ribasushi ok I was responding to:
But if Filecoin isn't then we can keep it here. |
There was a problem hiding this comment.
Thank you @Jorropo, did a quick first pass, details inline but quick feedback:
- This IPIP introduces many orderings without making it clear why we need all of them.
- Either simplify (remove magical fallbacks – maybe we can remove some types too?),
- or document when a client would want to use each of them and not the other ones.
- Cosmetic: lowercase and shorten ordering parameter and its values (some ideas below)
- Reuse preexisting RFC HTTP semantics for things like accepting multiple response types and listing supported ones (details around
Acceptin comments below)
| @@ -1,68 +0,0 @@ | |||
| # IPIP 0000: InterPlanetary Improvement Proposal Template | |||
There was a problem hiding this comment.
@Jorropo please restore the template 🙏 
There was a problem hiding this comment.
I thought this happened when an IPIP number was attribued (so before merging).
Anyway ok I'll
| 1. Blocks can arrive out of order, meaning when a block is consumed (data is red and returned to the consumer) and when it's received might not match. | ||
| 1. Blocks can be reused multiple times, this is handy for cases when you plan to cache on disk but not at all when you want to process a stream with use & forget policy. | ||
|
|
||
| What we really want is for the gateway to help us a bit, and give us blocks in a useful order. |
There was a problem hiding this comment.
@Jorropo Is this the proper characterization of the desired end goal? Or do we want to support other use cases (not clear from the document):
| What we really want is for the gateway to help us a bit, and give us blocks in a useful order. | |
| What we really want is for the gateway to help us a bit, and give us blocks in an order that removes the requirement of managing a block cache. |
| The block order correspond to someone reading the car file `Data` section (see car specification) from low index to high indexes. | ||
| In other words, blocks are sent first when streaming the car file over a fifo byte stream such as TCP connection, unix pipe, ... they are readable first. | ||
|
|
||
| `BlockOrder`, `enum (string)`, defaults to `Unordered` |
There was a problem hiding this comment.
This change will most likely be requested by IANA anyway, if we ever submit this as an update to existing https://www.iana.org/assignments/media-types/application/vnd.ipld.car, so better to clean this up from the beginning:
- Media type's optional parameters are lowercase, and shorter is better.
- 👉 Rename
BlockOrdertoorder, use lowercase for all values too (e.g.FooBarBuz→foo-bar-buz)
- 👉 Rename
There was a problem hiding this comment.
ok I thought we could put anything there. I'll
| For example: | ||
|
|
||
| - `Accept: application/vnd.ipld.car; BlockOrder=ForwardDFSWithSkippingOfDuplicates` means it is intrested in either one of: | ||
| - `ForwardDFSWithSkippingOfDuplicates` | ||
| - `ForwardDFSWithSkippingOfDuplicates-OnlyUsefull` | ||
| - `ForwardDFSWithDuplicates` | ||
| - `ForwardDFSWithDuplicates-OnlyUsefull` | ||
| - `Accept: application/vnd.ipld.car; BlockOrder=Unordered` would accept anything. | ||
|
|
||
| Clients may qualify it multiple times, for example | ||
| `Accept: application/vnd.ipld.car; BlockOrder=ForwardDFSWithSkippingOfDuplicates-OnlyUsefull; BlockOrder=Unordered` | ||
| in this case this is a list of preferences (lower indexes is better), the client would really like `ForwardDFSWithSkippingOfDuplicates-OnlyUsefull` | ||
| however if this isn't available it would accept `Unordered` (so anything) instead. | ||
|
|
||
| Clients SHOULD NOT ask for an Ordering that already covered by a previous ask, for example: | ||
| `Accept: application/vnd.ipld.car; BlockOrder=ForwardDFSWithSkippingOfDuplicates; BlockOrder=ForwardDFSWithDuplicates` | ||
| is illegal because `ForwardDFSWithDuplicates` is already compatible with `ForwardDFSWithSkippingOfDuplicates`. | ||
|
|
||
| In case a gateway sees this it should read this left to right and ignore already covered orderings (it can try matching it multiple times, but this would do nothing). |
There was a problem hiding this comment.
HTTP RFC 9110, 12.5.1. Accept already defines how to specify an HTTP request is accepting multiple response types:
Accept: application/vnd.ipld.car;order=foo;q=0.9, application/vnd.ipld.car;order=bar;q=0.8, application/vnd.ipld.car
We should simplify this spec, remove magical implicit fallbacks: format decision belongs to the client, server should not try to be smart and give "next best thing" – only fulfill request as-is.
There was a problem hiding this comment.
I didn't knew Accept worked this way.
Yeah definitely better.
|
|
||
| #### Ordering Incompatiblity | ||
|
|
||
| If none of the ordering requested by the client is supported the gateway MUST answer with `415 Unsupported Media Type`. |
There was a problem hiding this comment.
nit: add this code to https://github.com/ipfs/specs/blob/main/http-gateways/PATH_GATEWAY.md#response-status-codes + link to it.
I suggest we include note from https://httpwg.org/specs/rfc9110.html#status.415:
The
Acceptresponse header field can be used to indicate which media types would have been accepted in the request.
Sidenote: i believe this is how what you wanted to do with order=query should work.
There was a problem hiding this comment.
yes it is, and if it's how HTTP is supposed to work then it's better to follow this.
| ### `Query` | ||
|
|
||
| This order MUST NOT be suppported by the gateway. | ||
|
|
||
| This is the canonical way to trigger a failure and thus query which orderings are supported. | ||
|
|
There was a problem hiding this comment.
I quess if you want to have dedicated query convention, it could be 415 Unsupported Media Type HTTP error and Accept header:
| ### `Query` | |
| This order MUST NOT be suppported by the gateway. | |
| This is the canonical way to trigger a failure and thus query which orderings are supported. | |
| ### `Query` | |
| This order MUST NOT be suppported by the gateway. | |
| This is the canonical way to trigger a failure and thus query which orderings are supported. | |
| Returned HTTP Error MUST have supported orderings listed in the `Accept` response header field. | |
| Lack of error, or a response with `Content-Type` without `order` means `order` parameter is not supported by the gateway. |
That being said, having a dedicated fake-order type is an unnecessary spec complication, but against how content negotiation works in HTTP.
HTTP client can send multiple orderings in Accept with the request already (comma-separated), and even attach q weight to each one of them:
Accept: text/html, application/xhtml+xml, application/xml;q=0.9, image/webp, */*;q=0.8
Accept: application/vnd.ipld.car;order=foo;q=0.9, application/vnd.ipld.car;order=bar;q=0.8, application/vnd.ipld.car
| This indicates the start of the list of supported orderings which is a `, ` list of ordering up until the next newline, which indicates the end of the list (anything past this line MUST be ignored by clients). | ||
|
|
||
| The list doesn't need to include implicitly compatible orderings. | ||
|
|
||
| Regex: | ||
| ``` | ||
| .*: ((((Forward|Backward)DFSWith(SkippingOf|out)Duplicates)|Unordered|ProvenUsefullFirst)(-OnlyUsefull)?)(, ((((Forward|Backward)DFSWith(SkippingOf|out)Duplicates)|Unordered|ProvenUsefullFirst)(-OnlyUsefull)?))*$ | ||
| ``` | ||
|
|
||
| Example: | ||
| ``` | ||
| BackwardDFSWithSkippingOfDuplicates is not a supported ordering, supported orderings are: ForwardDFSWithSkippingOfDuplicates-OnlyUsefull, ForwardDFSWithDuplicates-OnlyUsefull | ||
| ``` |
There was a problem hiding this comment.
The magical Regex should be removed.
It is fine to have an example of human-readable HTTP error body, but the actual list of supported orderings (parsable by clients) should live in Accept header returned with the error response.
There was a problem hiding this comment.
Because I'll move to use 415 and Accept headers, this parsing of the error body is not needed anymore.
| For gateways: | ||
|
|
||
| `Unordered` is the only MUST implement ordering. | ||
|
|
||
| - `ProvenUsefullFirst` | ||
| - `ForwardDFSWithSkippingOfDuplicates` | ||
| - `ForwardDFSWithDuplicates` | ||
|
|
||
| are SHOULD implement orderings. | ||
|
|
||
| - `BackwardDFSWithSkippingOfDuplicates` | ||
| - `BackwardDFSWithDuplicates` | ||
|
|
||
| are MAY implement orderings. |
There was a problem hiding this comment.
I see 5 types of ordering, but no rationale why we need them.
Which one will be used by a low memory IoT device which is fetching firmware update as a CAR?
Are there other use cases that require different orders and handling of duplicates?
lidel
changed the title
|
Way overrenginered and confusing, closing in favor of #412. |
No description provided.