Add rel="canonical" for search engines

[lidel]

Context: #48 (comment)

While openzim/mwoffliner#963 solves problem for new snapshots, it is still possible the script will be run against an old ZIM without the header.

There are two ways to solve this:

(A) HTML tag

Before adding to IPFS the script should check if root document contains the header, and if not manually add it to every document.

Note: This is pretty expensive check/step that needs to happen after/during unpacking of ZIM archive and includes potential mutation of original file. It may not be feasible for 650GB archive of english wikipedia, which already takes long time to build.

(B) HTTP header

Instead of mutating every page, we could configure address the problem at Nginx level. Search engines are crawling *.wikipedia-on-ipfs.org, so we could simply add the rel=canonical HTTP header:

$ curl -I https://en.wikipedia-on-ipfs.org/wiki/Vincent_van_Gogh.html                                                                                                                   130 ...ikipedia-mirror master
HTTP/2 200
Content-Type: text/html; charset=utf-8
Content-Length: 457709
Link: <https://en.wikipedia.org/wiki/Vincent_van_Gogh>; rel="canonical"

Given the fact that HTML tag will be added to newly created ZIM snapshots when openzim/mwoffliner#963 lands, HTTP header sounds like a good temporary measure.

The task here would be to come up with dynamic Nginx rule that adds Link with URL matching current one, but without .html and -on-ipfs.

[lidel]

I've updated description of this task and added (B) HTTP header which may be more feasible way to address this short term.

[kelson42]

@lidel We have re-generated the WPTR ZIM file, so such old files won't be longer provided by us soon.

[lidel]

I confirmed the header is present in the latest snapshot handled since #77:

  <link rel="canonical" href="https://tr.wikipedia.org/wiki/Anasayfa">

Let's continue in #60 #61