Week 03 2025 routine

[kiwixbot]

Check nodes free space

df -h / && df -h /data

• create a report in issue comment

Nodes system upgrades

apt update && apt upgrade

• run systematically the upgrade on bastion, stats, services, storage, demo, mirrors-qa nodes
• check for and apply important security upgrade on worker nodes asap (imager-worker, ondemand, sisyphus)

(regular workers updates are done separately on a monthly basis for worker nodes to not impact production)

Backups

• Ensure all borg repositories are being updated

k8s cluster

• Check Pod errors or in CrashLoopBackoff

k get pods -A -o wide|grep -E 'Error|Crash'

• Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• Upgrade k8s if applicable/possible

Stats

matomo - stats.kiwix.org

• Ensure download.kiwix.org stats are being recorded
• Check whether matomo should be upgraded

Grafana

• Alert list is normal
• Zimfarm dashboard is normal
• Jobs durations dashboard is normal
• There is no abnormal behaviors on cluster resources consumption
• All mdadm RAID arrays are OK (check all instances in the top left combobox)

Projects

• UptimeRobot has no alert
• zimit backlog is reasonable
• Cloud Code Signing Certificate usage is OK (look for Unused Signings under END ENTITY CERTIFICATES): we have 1,200 per year (August to August)
• Analyze zimit failed tasks and document bad domains in our WIP blacklist
• PRs awaiting your review

Security

• Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[rgaudin]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	17G	19G	49%	-
stats	/	233G	144G	77G	66%	+13G
services	/	456G	226G	207G	53%	+5G
storage	/	33T	9.7T	22T	32%	+0.2T
imager-worker	/	1.9T	281G	1.5T	16%	don't care
sisyphus	/	233G	20G	201G	-	don't care
ondemand	/	25G	14G	11G	56%	-
ondemand	/data	216G	202M	205G	1%	don't care
demo	/	40G	9.5G	28G	26%	-2G
demo	/data	1.8T	920G	743G	56%	don't care

storage node has a lot of packages pending upgrade ; since does include k8s-related ones, I'm leaving it for next monthly routine.

I did upgrade rsync in regard to the recent security issue but there's no running rsyncd on the host.

misc

• website backup cronjob was suspended… I resumed it
• library demo generator error #354
• restarted the CMS API
• Unused Signings: 852 (same as W1)
• zimit pipe is empty

zimit

• (2) net::ERR_NAME_NOT_RESOLVED (Seed Page Load Failed)
• (3) net::ERR_HTTP2_PROTOCOL_ERROR (Seed Page Load Failed)
• (1) net::ERR_ABORTED (Seed Page Load Failed)
• (1) net::ERR_INVALID_AUTH_CREDENTIALS (Seed Page Load Failed)
• (1) net::ERR_HTTP_RESPONSE_CODE_FAILURE (Seed Page Load Failed). Was HTTP 436 (doesnt exists)
• (15) Main URL returned an unprocessable HTTP code: 403
• (6) Main URL returned an unprocessable HTTP code: 404
• (1) Main URL returned an unprocessable HTTP code: 502
• (5) Navigation timeout of 90000 ms exceeded (Seed Page Load Failed)
• (1) memory consumption (exit code 137) openzim/warc2zim#383
• (3) LookupError: unknown encoding: unicode openzim/warc2zim#331 with: U, utf-2, UT
• (2) Rewriter choke on a relative URL inside a querystring openzim/warc2zim#380
• (3) multipart.ParserError: Unexpected end of multipart stream (parser closed) openzim/warc2zim#432
• (1) Missing title
• (1) died with Signals.SIGKILL openzim/zimit#455

We're getting more of those HTTP Protocol error. Is it what it looks like? ie: the crawler not able to work with HTTP/2 ? If so, we should maybe raise the priority of this to browsertrix.

[benoit74]

Regarding HTTP2, we already confirmed that HTTP2 is supported. To me it is just that more websites are upgrading to HTTP2 (often behind a proxy like Cloudflare).

[rgaudin]

OK ben if its supported, what is this issue reporting?

[benoit74]

From what I find on the web, this error reports that there has been an error inside the HTTP2 protocol (connection broken, dropped by server, ...). Probably because browser has been blocked by some kind of protection.

[rgaudin]

OK