Allow to store data as references in the store - hackish version

[rklaehn]

Actually not as hackish as expected. The provenance stuff is probably good to go with a few more tests.

Still, not quite sure how to proceed with this.

Do we increase the quality of this and merge it to main, meaning that we now implement experimental features of ipfs like the file store pattern / nocopy in addition to the ability to store stuff directly in the store, which we still need to retain for content from remote nodes?

Does this end up a deltachat exclusive?

  --nocopy                   bool   - Add the file using filestore. Implies raw-leaves. (experimental).

Implements n0-computer/beetle#14

[Frando]

Even though this adds some complexity and there certainly are a few challening integration and abstraction questions still to go through, I think this would be a great feature to have independently of Deltachat and iroh-share. Here's why: I regularily work with large collections of media files from community media publishers with limited infrastructure resources. We are working towards better tools to share and replicate these libraries. A nocopy mode makes this much simpler, faster and cheaper to try out with iroh. Copying 10-20TB of mp3s takes a long time and needs a lot of spare server capacity to try things out.

[Frando]

(maybe this fits better into an issue, but adding here for now)
I've been thinking about the tradeoffs between different approaches to sync local files trees with p2p file stores, both when I worked more with hyperdrive and also with IPFS, for the reasons mentioned above. There's several strategies:
a) store has file references (pro: cheap, con: universal mutability is hard)
b) copy from local fs to store (pro: simple, con: double storage, long import time, dealing with updates&conflicts)
c) all data in store plus FUSE for local fs integration (pro: allows tailored interfaces, con: creating a good and high perf impl is quite some work)
for a) and b) there can also be repo-like interfaces through dotfiles in any directory to allow a "log" and "commit" workflow from the local fs
In my experience, tradeoffs between the different approaches are real and matter in practice when working with larger existing file collections, especially a no-copy mode vs full copy, and fuse impls becoming a pain if not working well enough. So take this as a motivation to enable iroh to work well in constrained environments by enabling different storage patterns to evolve for different situations :)

[ramfox]

Hello! The code that was previously hosted in this repository has been moved to n0-computer/beetle. beetle is in maintenance mode, but will still accept PRs.

If you are still interested in getting your PR merged, please re-open your PR on n0-computer/beetle.

Check out our blog post for more info on our new direction for iroh: https://n0.computer/blog/a-new-direction-for-iroh/