Online Preview (ZIM reading over HTTP)

[kelson42]

I propose to add a new feature to Kiwix JS, the capacity to ready a ZIM file online via HTTP. It should provide to the user a similar UX than with an offline/local ZIM file.

This feature would allow to:

• Offer a preview to users before download of the full ZIM file
• Is a necessary first step before implementing an IPFS ZIM file reader(see kiwix-js + IPFS = unstoppable wikipedia mirror ipfs/devgrants#49)
• Allow to evaluate an alternative to kiwix-serve approach
• ...

From the user perspective the workflow could look like following: In "Configure" Tab, below the "Browser" button to load local ZIM file, we would have a text field to enter a HTTP URL pointing to a ZIM file online. This online ZIM file would be then loaded like (from a user perspective) a local file one time the user have clicked on the "Go" button beside the URL input text.

For this first implementation of a JS ZIM reader over HTTP, the page loading speed is not critical, it can be slow in a way this is still usable. If the loading takes a bit of time then a progress bar or similar should be implemented.

The key/blocker points are:

• agnostic access to any ZIM file online via HTTP
• Kiwix JS works in a similar manner when offline/online ZIM
• Not extra manipulation needed on the server side

Remark: This ticket is a follower of #595

[Jaifroid]

Just to add that, as discussed in #595, merging #658 would be a step towards this, given that caching is essential for dirEntry binary search to attain an acceptable speed over a network or with slow file access.

[Jaifroid]

It will be important to ensure that the server accepts the Range header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests .

[mossroy]

This proposal needs more thinking IMHO.

First, we need to keep in mind that we need to replace our custom javascript backend implementation with an emscripten compiled version of libzim/kiwix-lib. In their current forms, these libraries do not support reading ZIM files through HTTP.

If we implement such "over HTTP" feature in kiwix-js, it will become an issue when switching to libzim : either the feature disappears from kiwix-js (and the forks relying on it will have to keep an outdated version) or we will have to implement it in libzim (maybe with https://emscripten.org/docs/api_reference/fetch.html, but it has to be discussed and tested with their maintainers).

I'm also puzzled by the fact to give this preview feature to the end-users.
It's already complicated to make first-time users understand they have to download a ZIM file. If we give them an option to read ZIM files through HTTP, they'll probably pick this much simpler option.
And this becomes really confusing for them : they might think "what's the point of an offline reader if it needs internet access?"
Add the fact that we already know it will be very slow (even with #658 merged), and we end up with a terrible user experience.

From what I read in #595, this won't be usable without additional pre-computed indexes (probably inside the ZIM file) : shouldn't this be discussed with libzim maintainers first? Else I don't see the point.

Don't get me wrong : I understand this could be really helpful for the IPFS feature and/or for using some ZIM files from a local network. But we should not deploy some feature in kiwix-js unless we know it is sustainable in the mid/long term, and does not confuse end-users

[kelson42]

First, we need to keep in mind that we need to replace our custom javascript backend implementation with an emscripten compiled version of libzim/kiwix-lib. In their current forms, these libraries do not support reading ZIM files through HTTP.

If we implement such "over HTTP" feature in kiwix-js, it will become an issue when switching to libzim : either the feature disappears from kiwix-js (and the forks relying on it will have to keep an outdated version) or we will have to implement it in libzim (maybe with https://emscripten.org/docs/api_reference/fetch.html, but it has to be discussed and tested with their maintainers).

Pertinent, but without visibility on libkiwix emscripten, I assume this won't happen soon.

I'm also puzzled by the fact to give this preview feature to the end-users.
It's already complicated to make first-time users understand they have to download a ZIM file. If we give them an option to read ZIM files through HTTP, they'll probably pick this much simpler option.
And this becomes really confusing for them : they might think "what's the point of an offline reader if it needs internet access?"
Add the fact that we already know it will be very slow (even with #658 merged), and we end up with a terrible user experience.

Yes, but all of this seems a problem easier to fix than the technical ones. In worse case we can make another project with another name to avoid confusion.

From what I read in #595, this won't be usable without additional pre-computed indexes (probably inside the ZIM file) : shouldn't this be discussed with libzim maintainers first? Else I don't see the point.

You are right that a preindex needs a discussion at the ZIM level (in repository openzim/libzim or openzim/overview). This ticket requires it works out-of-the-box with ZIM files like they are today (I have expressed clearly my point of view here: #595 (comment)). Developers who don't believe it could work, I guess, are not going to implement it (I understand this is now the hard part) :)

Don't get me wrong : I understand this could be really helpful for the IPFS feature and/or for using some ZIM files from a local network. But we should not deploy some feature in kiwix-js unless we know it is sustainable in the mid/long term, and does not confuse end-users

Your comment is 100% pertinent to me.

[Jaifroid]

I think the thing that interests me most about this ticket is the possibility of enabling easy access to a very large ZIM over an Intranet / LAN, without much overhead. But I've no idea how viable it is really, until the various pieces of the puzzle come together!

[mossroy]

Regarding the user experience, another way to handle that would be to hide this feature in the UI, and have a "secret" way to show it. This would allow to keep only one project, one source code, and one deployment.

In this case, I would see that as a "playground" feature for now, that is not ready for end-users, and could be removed in the future (either because it's not viable, or because the switch to libzim breaks it)

If everybody is OK with that (including @javgh and @ikreymer), I'm OK too.

NB : I won't have time to work on that myself. If I find some time, I'll work on the libzim emscripten feature

[Jaifroid]

Nice! We could have our very own kiwix://flags URL to type into the search box, which would allow an advanced user to enable/test this as an experimental option...

[Jaifroid]

@kelson42 Is it still worth keeping this issue open? You use Kiwix Serve with library.kiwix.org to enable preview, and that seems to work fine. Also, implementing this (if possible) in Kiwix JS would mean no preview for Zimit archives currently. I'm closing but feel free to re-open if this is still wanted/needed.