Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Content claims locator fetches ALL shards before data, making large directories slow #302

Open
hannahhoward opened this issue Jan 22, 2025 · 0 comments

Comments

@hannahhoward
Copy link
Member

hannahhoward commented Jan 22, 2025

What

When fetching data from a large shard, the content claims locator fetches the index, then it fetches a location claim for every single shard and it does so BEFORE it ever fetches data from any of the shards. This means extract a small amount of data from a large shard is very slow.

Instead, it only needs to initially fetch the shard that contains first CID to fetch (i.e. the digest that was used calling locate).

Attached trace screenshot:

Image

@hannahhoward hannahhoward converted this from a draft issue Jan 22, 2025
@hannahhoward hannahhoward moved this from Sprint Backlog to In Progress in Storacha Project Planning Jan 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

No branches or pull requests

1 participant