steamcache/monolithic for ARM based devices

[KiaArmani]

The previous steamcache/steamcache repository works great with ARM-based devices (e.g. ODROID HC2 (has Gigabit LAN and a SATA-port, making it feasible for this service), but I am unable to get steamcache/monolithic to run on it because of the notorious exec error:

standard_init_linux.go:190: exec user process caused "exec format error"

This is solved by manually building the repo for steamcache/steamcache, but doesn't work for steamcache/monolithic. Is there any way I can get this to run on ARM?

Sincerely,
Kia.

[thazlett]

Watching.

I used the old steamcache container on a rock64 with a USB3 attached SSD. It's surprising how well this thing works for such a small / low powered device. It manages 500mbps quite happily, sometimes more.
I had to build the container after switching out the base image to an ARM one(same with the dns one).

The monolithic one appears to have several containers/layers. But I managed to get this to work on ARM by tweaking the the base one. I am now using it on the same rock64.

How?

Under the base ubuntu container, changed to base to 'arm64v8/ubuntu:bionic'. Then I swapped out ubuntu/overlay/etc/apt/sources.list with:

deb http://ports.ubuntu.com/ bionic main restricted universe multiverse
deb http://ports.ubuntu.com/ bionic-security main restricted universe multiverse
deb http://ports.ubuntu.com/ bionic-updates main restricted universe multiverse
deb http://ports.ubuntu.com/ bionic-backports main restricted universe multiverse

This then allowed the other containers to build on my rock64.

Why did that work?

The ubuntu container it's based on appears to try and fetch repo info from a repo that does not exist for arm64. It's on this apt update stage when building the nginix container where it fails due to 404s:
Step 6/8 : RUN apt-get update && apt-get install -y nginx inotify-tools ---> Running in 92c325990e9e Get:1 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic InRelease [242 kB] Get:2 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] Get:3 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] Get:4 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/main Sources [1,063 kB] Get:5 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/universe Sources [11.5 MB] Get:6 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/restricted Sources [5,823 B] Ign:7 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/restricted arm64 Packages Ign:8 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/universe arm64 Packages Ign:9 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/main arm64 Packages Get:10 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/restricted Sources [2,098 B] Get:11 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/main Sources [331 kB] Get:12 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/universe Sources [194 kB] Ign:13 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/universe arm64 Packages Ign:14 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/main arm64 Packages Ign:15 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/restricted arm64 Packages Get:16 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/universe Sources [42.8 kB] Get:17 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/restricted Sources [1,483 B] Get:18 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/main Sources [94.0 kB] Get:19 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/multiverse Sources [2,296 B] Ign:20 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/multiverse arm64 Packages Ign:21 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/main arm64 Packages Ign:22 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/restricted arm64 Packages Ign:23 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/universe arm64 Packages Ign:7 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/restricted arm64 Packages Ign:8 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/universe arm64 Packages Ign:9 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/main arm64 Packages Ign:13 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/universe arm64 Packages Ign:14 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/main arm64 Packages Ign:15 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/restricted arm64 Packages Ign:20 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/multiverse arm64 Packages Ign:21 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/main arm64 Packages Ign:22 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/restricted arm64 Packages Ign:23 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/universe arm64 Packages Ign:7 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/restricted arm64 Packages Ign:8 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/universe arm64 Packages Ign:9 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/main arm64 Packages Ign:13 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/universe arm64 Packages Ign:14 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/main arm64 Packages Ign:15 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/restricted arm64 Packages Ign:20 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/multiverse arm64 Packages Ign:21 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/main arm64 Packages Ign:22 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/restricted arm64 Packages Ign:23 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/universe arm64 Packages Err:7 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/restricted arm64 Packages 404 Not Found [IP: 212.219.56.184 80] Ign:8 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/universe arm64 Packages Ign:9 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic/main arm64 Packages Err:13 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/universe arm64 Packages 404 Not Found [IP: 212.219.56.184 80] Ign:14 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/main arm64 Packages Ign:15 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-updates/restricted arm64 Packages Err:20 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/multiverse arm64 Packages 404 Not Found [IP: 212.219.56.184 80] Ign:21 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/main arm64 Packages Ign:22 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/restricted arm64 Packages Ign:23 http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu bionic-security/universe arm64 Packages Fetched 13.7 MB in 12s (1,114 kB/s) Reading package lists... E: Failed to fetch http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu/dists/bionic/restricted/binary-arm64/Packages 404 Not Found [IP: 212.219.56.184 80] E: Failed to fetch http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu/dists/bionic-updates/universe/binary-arm64/Packages 404 Not Found [IP: 212.219.56.184 80] E: Failed to fetch http://mirrorservice.org/sites/archive.ubuntu.com/ubuntu/dists/bionic-security/multiverse/binary-arm64/Packages 404 Not Found [IP: 212.219.56.184 80] E: Some index files failed to download. They have been ignored, or old ones used instead. The command '/bin/sh -c apt-get update && apt-get install -y nginx inotify-tools' returned a non-zero code: 100
I use ARM/ubuntu based containers for a number of things so this seemed a little odd to me. Then I tracked down where 'mirrorservice' was coming from.

[KiaArmani]

I'm setting up a fork over @ https://github.com/KiaArmani/ubuntu with your suggested changes and will close this issue when it worked. Thanks a lot!

[Fidenz-Lasithh]

@KiaArmani did you get it to work?

[KiaArmani]

Unfourtunately no, since my device is not 64-bit. Von: Lasith Hettiarachchi <notifications@github.com> Gesendet: Samstag, 27. April 2019 11:47 An: steamcache/monolithic <monolithic@noreply.github.com> Cc: KiaArmani <kiaarmani@web.de>; Mention <mention@noreply.github.com> Betreff: Re: [steamcache/monolithic] steamcache/monolithic for ARM based devices (#21) @KiaArmani <https://github.com/KiaArmani> did you get it to work? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4CKN7BYS2ADEPMLQ74BZLPSQOIHANCNFSM4G4F474A> . <https://github.com/notifications/beacon/AA4CKNYYNGRFG7DEM5JWFDDPSQOIHANCNFSM4G4F474A.gif>

[VibroAxe]

We just made some changes to the downstream ubuntu images, does this help people?

We don't have any intention of maintaining arm images at the moment, but are happy to try and add little things to improve efforts!

[pheetr]

@VibroAxe , I have a Raspberry Pi 4 4G back ordered atm, on which I intend to run this at a party with ~6 people in a few weeks.
I expect to have the hw in about 5-10 days, after which I'll provide feedback ... and possibly cry for help as I'm a Docker novice :)

[VibroAxe]

@pheetr interesting. We don't currently recommend running lancache on a RPI for performance reasons, but for a small 6 person Lan party I'd be very interested to see what performance you see out of it

[pegasy]

@pheetr Did you get your pi yet?
I have been trying to get this to run on my Pi 4 with this script to clone + build all the layers for monolithic, that part works just fine but when I try to start it it failes at Checking nginx config

See this gist for a log of the output I get, I don't know if its RPI related or not sadly, I have never used steamcache / monolithic before so might have messed up somewhere, I ran the following commands to try to bring it up but monolithic just keeps restarting :/

export HOST_IP=`hostname -I | cut -d' ' -f1`
docker run --restart unless-stopped --name lancache-dns --detach -p 53:53/udp -e USE_GENERIC_CACHE=true -e LANCACHE_IP=$HOST_IP pegasy/lancachenet_lancache-dns:latest
docker run --restart unless-stopped --name lancache --detach -v /cache/data:/data/cache -v /cache/logs:/data/logs -p 80:80  pegasy/lancachenet_monolithic:latest
docker run --restart unless-stopped --name sniproxy --detach -p 443:443 pegasy/lancachenet_sniproxy:latest
echo Please configure your router/dhcp server to serve dns as $HOST_IP

[VibroAxe]

Looks like the RPI doesn't have enough cache memory for the default config

nginx: [emerg] could not build map_hash, you should increase map_hash_bucket_size: 32

Try a value of either 64 or 128
map_hash_bucket_size 32|64|128;

You should be able to make this work by creating a new file in overlay/etc/nginx/conf.d/ and rebuilding the image

[pheetr]

@pheetr interesting. We don't currently recommend running lancache on a RPI for performance reasons, but for a small 6 person Lan party I'd be very interested to see what performance you see out of it

@VibroAxe Yeah, I wouldn't have tried it with the Pi3, but the Pi4 with its beefier CPU, more ram, USB3 (SSD) and GbLAN should hopefully cope. :)

@pegasy I've received shipment confirmation on Friday, so I'm hopeful that I'll have it by Tuesday the latest. I'm glad I'm not the only one with this idea. This gives me even more confidence that we'll get it to work, hehe.

[pegasy]

Added this file

pi@rpi4b-1:~/git/lancachenet_rpi $ cat ../monolithic/overlay/etc/nginx/conf.d/15_set_map_hash_bucket_size.conf
map_hash_bucket_size 128;

Then I got another error

Checking nginx config
nginx: [emerg] "proxy_max_temp_file_size" directive invalid value in /etc/nginx/sites-available/generic.conf.d/root/20_cache.conf:29
nginx: configuration file /etc/nginx/nginx.conf test failed
ERROR: hook /hooks/supervisord-pre.d/99_config_check.sh} returned a non-zero exit status '0'

Found this thread for nextcloud that had the same error, seems like we would be limited to below 2048m files (I tried with 1920m and that worked), really not sure why filesize is tied to it being 32bit os, hopefully someone can try this on a 64bit os on rpi.

Everything seems to be working, just need to figure out some stuff with the dns I think since not all requests seems to go via the cache (I tried downloading the same game over and over, each time the cache grew a bit and install time went down).

Video of downloading a cached game: https://youtu.be/92SAgDNKumc with htop, iftop and router traffic

My goal was to speed up downloads when me and my SO downloads the same game (yay coop) or steam workshop content since we only have a 4g connection where we get 20-40 mbit (luckily no data caps), but when both wanna download a few GB it does take some time.

Would love to see if anyone try it with more clients using it how it holds up :)

[MattzFiber]

Can confirm that @pegasy's scripts and configs work well.
I only have an RPI3, and while using an SSD on USB and ethernet, the USB BUS becomes the bottleneck.
To try and work around this, I tried to offload the network traffic to WLAN 5Ghz, yet I was totally unable to get more than 30-50mbps on an initial download.
Once it was cached, I could see spikes to 120mbps for a re-download, although this data may still have been in RAM.
Can say for sure its not worth it on an RPI3B+. However, with the huge changes on the RPI4, it seems like that might be a better option.

For a home user with a few PCs trying to save on some bw or increase throughput, it might be worth exploring.

Thanks for the compile script @pegasy

[MattzFiber]

Skip the above, looks like I couldn't get any higher perfoamnce when moving to an x64 host with SSDs direct attached either, seems to hit the same performance. Looks like its related to #16

[jrcichra]

I used docker's buildx cross compile in Github Actions to build all the lancache containers for the Pi. My repo also has the docker-compose you can use: https://github.com/jrcichra/lancache-rpi

[unspec]

I've tagged this for-reference and will close as no specific outstanding questions. Feel free to reopen if there are any specific 'upstream' improvements that would aid ARM deployments or more useful tips.