Apt-Cacher-NG LXC

[tteck]

To enable Apt-Cacher-NG for your existing containers, execute (after configuring the <apt-cacher-ng-IP>) the following command in the console of each LXC that you wish to use with the proxy. Update #2332 (comment)

echo "Acquire::http { Proxy \"http://<apt-cacher-ng-IP>:3142\"; };" >/etc/apt/apt.conf.d/00aptproxy

The scripts now have the option to utilize Apt-Cacher-NG (Advanced settings) when creating LXCs. The added functionality is expected to decrease bandwidth usage and expedite package installation and updates.

[tteck]

Apt-Cacher-NG is incredibly helpful for me. For instance, if John Doe1 reports an issue with the Debian 12 LXC, the first step is to create a Debian 12 LXC. During this process, I cache standard dependencies like curl, sudo, and midnight commander that are used by every script. After that, It updates the Debian 12 OS packages, which are now cached.

Now, if John Doe2 reports an issue with the Docker LXC, which utilizes Debian 12, I create a new Docker LXC. However, this time, I locally serve the cached packages that were cached during the creation of the Debian 12 LXC, saving bandwidth and a faster package installation.

Another example is, let's say we have 15 LXCs created, and we need to update the OS. If we use the Proxmox VE LXC Updater, it updates the first LXC while caching the packages. Subsequently, the other 14 LXCs can locally utilize those cached packages.

[rloomans]

A slightly easier way to configure Debian or Ubuntu to use a package cache is to install the auto-apt-cache package ( https://salsa.debian.org/debian/auto-apt-proxy). It requires an additional DNS record for your local network, but you only need to do that once.

…

On Sat, 13 Jan 2024 at 06:29, tteckster ***@***.***> wrote: To enable Apt-Cacher-NG for your existing containers, execute (*after configuring the IP*) the following command in the console of each LXC that you wish to use with the proxy. echo "Acquire::http { Proxy \"http://<apt-cacher-ng IP>:3142\"; };" >/etc/apt/apt.conf.d/00aptproxy — Reply to this email directly, view it on GitHub <#2332>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAWPZJYXMDFGKUGWURHKATYOGMKHAVCNFSM6AAAAABBYVMOUGVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWGA3DONRSG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[tteck]

Why is that considered easier?

[Solear]

It requires an additional DNS record for your local network, but you only need to do that once.
How? I have forwarding TCP Port 3142 from my gateway to the IP of the apt-cacher-ng lxc. But it doesnt work. Or is there a way to reroute. this via adguard?

[1oh1]

@Solear you need an SRV record pointing to apt-cacher-ng. It looks like it's possible in Adguard. You can refer to this Cloudflare blog post to learn more about SRV records.

Assuming your local domain is example.com and the hostname for the apt-cacher-ng container is apt-cacher-ng.example.com, the record might look like

||_apt_proxy._tcp.example.com^$dnsrewrite=NOERROR;SRV;0 0 3142 apt-cacher-ng.example.com

OR

||_apt_proxy._tcp.apt-cacher-ng.example.com^$dnstype=SRV,dnsrewrite=NOERROR;SRV;0 0 3142 example.com

I guess you'll need a DNS rewrite as well to resolve apt-cacher-ng.example.com to the IP of your apt-cacher-ng container. Maybe these will help:

1. How do I set up SRV for local services? AdguardTeam/AdGuardHome#5977
2. https://medium.com/@life-is-short-so-enjoy-it/homelab-adding-local-dns-entry-into-adguard-home-arpa-and-pushing-to-clients-from-udm-se-8493253830e5
3. https://medium.com/@life-is-short-so-enjoy-it/homelab-adguard-adding-local-dns-entries-dns-rewrite-vs-customer-filtering-rules-ca8e4992a21e

[remz1337]

It's easier than that. Simply create a AAAA record in your DNS resolver. I added a DNS name record in pi-hole and everything works with auto-apt-proxy.

In Pi-hole --> Local DNS --> DNS record, add domain apt-proxy pointing to the IP of your AptCacherNg machine

[rloomans]

I find adding the package (with apt install auto-apt-proxy) easier to remember and type, I can do it the same way in docker images, lxc, vms or bare metal hosts. Bonuses are: - if I change my package cache host/port/etc, I can just change the DNS record and everything picks it up - the script that does the auto discovery tests that the cache is up before returning the proxy details, so if it’s not, there is a slight delay and apt continues connecting to the repositories.

…

On Sat, 13 Jan 2024 at 23:13, tteckster ***@***.***> wrote: Why is that considered easier? — Reply to this email directly, view it on GitHub <#2332 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAWPZJ43G577YIEROYGOG3YOKB55AVCNFSM6AAAAABBYVMOUGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DCMJXHEZDM> . You are receiving this because you commented.Message ID: ***@***.***>

[manuelkamp]

I found out that I cover 99% of the traffic with the standard apt repos, so for example if I configure manually another repo on one LXC/VM it does not make sense to cache it. Therefore I have this line in my post-deployment script which does the job for me:
sed -i 's/http:\/\//http:\/\/srv-apt-cache.somelocaldomain.local:3142\//g' /etc/apt/sources.list

[rdlu]

Nice! It was easy to add!

I just had an issue with https repos, specifically the postgres ones.

I adjusted the /etc/apt-cacher-ng/acng.conf with PassThroughPattern: .*

The official script here adds a comment after, but somehow it's interfering. Removing the comment makes the postgres https repos that I added to a db lxc I have functional again.

[tteck]

Tweaked the sed command to remove the comment also. 87b7fc7

[Manichee]

I had this problem installing tonight. No comment after the line, but seemed to be an extra space at the end that would cause https to not work. I deleted the space at the end of the line and https repos work now.

[tteck]

Removed space in sed command e6e2b98

[tikismoke]

Is it possible to have a sort of env_var on the host that can be used by any script automatically?

[tteck]

Could you provide further clarification or elaborate on that?

[tteck]

Presently, when you opt for Apt-Cacher-NG in Advanced settings during the creation of LXCs, it automatically appends the following line to the LXC being created: echo "Acquire::http { Proxy \"http://<apt-cacher-ng IP>:3142\"; };" >/etc/apt/apt.conf.d/00aptproxy.

[tteck]

If you choose to discontinue the use of Apt-Cacher-NG, you can execute the command rm /etc/apt/apt.conf.d/00aptproxy in the LXC console.

[Manichee]

This may not be the place for it, but I have an issue with this when trying to use this lxc on other debian containers. If the container is ubuntu it works fine:

Operating System: Ubuntu 22.04.3 LTS 
# apt update
Hit:1 https://packages.wazuh.com/4.x/apt stable InRelease
Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.

However on debian containers I get:

Operating System: Debian GNU/Linux 12 (bookworm)  
# apt update
Ign:1 http://deb.debian.org/debian bookworm InRelease
Ign:2 http://security.debian.org bookworm-security InRelease                                                        
Ign:3 http://deb.debian.org/debian bookworm-updates InRelease                                                       
Get:4 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  InRelease [1477 B]                             
Hit:5 https://download.docker.com/linux/debian bookworm InRelease                                             
Ign:1 http://deb.debian.org/debian bookworm InRelease
Ign:2 http://security.debian.org bookworm-security InRelease
Ign:3 http://deb.debian.org/debian bookworm-updates InRelease
Ign:2 http://security.debian.org bookworm-security InRelease
Ign:1 http://deb.debian.org/debian bookworm InRelease
Ign:3 http://deb.debian.org/debian bookworm-updates InRelease
Err:1 http://deb.debian.org/debian bookworm InRelease                                                               
  503  Resource temporarily unavailable [IP: 192.168.1.97 3142]
Err:2 http://security.debian.org bookworm-security InRelease                                                        
  503  Resource temporarily unavailable [IP: 192.168.1.97 3142]
Err:3 http://deb.debian.org/debian bookworm-updates InRelease                                                       
  503  Resource temporarily unavailable [IP: 192.168.1.97 3142]

[tteck]

No issues here with the Debian bookworm repositories

deb http://deb.debian.org/debian bookworm main contrib
deb http://deb.debian.org/debian bookworm-updates main contrib
deb http://security.debian.org bookworm-security main contrib

[Manichee]

So figured it out. Seems I had a DNS issue on the container. Once I set the DNS servers it started working for the most part.

[techie2000]

I had that error earlier today on the debian based lxc's (both v11 and v12). I increased memory/swap on the apt-cahcher-ng lxc from 512mb to 2gb and the error went away.

[tteck]

For anyone wanting to switch to the new method introduced here ffd2fc3, change <apt-cacher-ng-IP> to your proxy IP address, then execute the snippet within the LXC console.

echo "Acquire::http::Proxy-Auto-Detect \"/usr/local/bin/apt-proxy-detect.sh\";" >/etc/apt/apt.conf.d/00aptproxy
cat <<EOF >/usr/local/bin/apt-proxy-detect.sh
#!/bin/bash
if nc -w1 -z <apt-cacher-ng-IP> 3142; then
echo -n "http://<apt-cacher-ng-IP>:3142"
else
echo -n "DIRECT"
fi
EOF
chmod +x /usr/local/bin/apt-proxy-detect.sh

It will overwrite your previous /etc/apt/apt.conf.d/00aptproxy

Discussions

[Citroene]

is there a way to delete the proxy again?

[tteck]

rm /etc/apt/apt.conf.d/00aptproxy /usr/local/bin/apt-proxy-detect.sh
When you execute the above command, both specified files (00aptproxy and apt-proxy-detect.sh) will be permanently deleted from the filesystem.

[remz1337]

I think there is an even easier way to detect/add the proxy to your LXC, using avahi. Simply install the squid-deb-proxy-client package. It auto detects the AptCacherNG server and configures the LXC accordingly. Automagic :)

First, setup the AptCacherNG server (Ubuntu 22.04), with apt install apt-cacher-ng avahi-daemon (as per these instructions)

Then on any Debian/Ubuntu LXC that you want to use the cache server, run apt install -y squid-deb-proxy-client

[ianalexis]

Up

[1oh1]

auto-apt-proxy wasn't resolving the SRV record for me since I had an incorrect Search domain in my Proxmox node's DNS settings (I was using my external .com domain instead of the internal localdomain)

Datacenter > node > DNS

Now it matches the system domain configured in OPNSense and I also had Unbound configured to register DHCP leases so any new services that spin up and use DHCP get registered in Unbound as <service-hostname>.localdomain)

To create the SRV record, since I'm using OPNSense I installed the routerperformance OPNSense repo and installed Unbound Custom Options (os-unboundcustom-maxit) plugin and added this under Services > Unbound DNS > Custom Options

server:
local-data: "_apt_proxy._tcp.localdomain 86400 IN SRV 0 0 3142 apt-cacher-ng.localdomain."

Restarted Unbound DNS service and tested that the SRV record is resolved with dig:

❯ dig _apt_proxy._tcp.localdomain. SRV

; <<>> DiG 9.11.9 <<>> _apt_proxy._tcp.localdomain. SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19157
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_apt_proxy._tcp.localdomain.   IN      SRV

;; ANSWER SECTION:
_apt_proxy._tcp.localdomain. 86400 IN     SRV     0 0 3142 apt-cacher-ng.localdomain.

;; Query time: 4 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sun Mar 17 10:33:34 IST 2024
;; MSG SIZE  rcvd: 101

Connect to the shell of any Proxmox container created using the helper scripts and install auto-apt-proxy and test that the SRV record is resolved:

root@vscode-server:~# hostname --fqdn  
vscode-server.localdomain
root@vscode-server:~# apt install auto-apt-proxy -y && auto-apt-proxy
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
auto-apt-proxy is already the newest version (14.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
http://apt-cacher-ng.localdomain:3142

Now running update will use the APT proxy without having to explicitly define it during container creation or after. I couldn't install squid-deb-proxy-client because I guess it's deprecated

root@vscode-server:~# apt install squid-deb-proxy-client
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package squid-deb-proxy-client is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'squid-deb-proxy-client' has no installation candidate

[remz1337]

Hey, I think there is an easier solution. See my other comment here:
#2332 (reply in thread)

[jcastro]

Hi! I think it would be helpful to have the setup and removal of the apt-catcher in the website itself! I wasn’t sure how to use it or enable it until I took a look at this :). In any case, thanks a lot!

[tteck]

If there's a Post Install button, additional information will be shown (this page).

[jcastro]

Hey! Another question. I've executed this command on all my LXCs.

echo "Acquire::http { Proxy \"http://<apt-cacher-ng-IP>:3142\"; };" >/etc/apt/apt.conf.d/00aptproxy

I've already had some updates on the LXCs, so I wanted to either see some logs or cached files or something to know about the apt-cacher-ng. Checking their help page and looking at /var/log/apt-cacher-ng, there are no log files.

On the admin page, I have these stats!

[hastmu]

So maybe it is helpful, but please decide on your own.

I run also squid-deb-proxy and apt-cacher-ng, since years and configure that for every of my setups (debian systems), via a custom deb package. I had also struggles with some topics you discuss here, like easy config and that sometimes the acl of the proxy does not allow some sources and i need a direct declaration for that.

For solving all that i did: https://github.com/hastmu/apt-proxy-detect

What do you get?

• Proxy detection via
  • avahi
  • set defaults
• caching of URL specific working proxy (or none if none works)
• caching over reboot if possible
• allow mixture of allowed and non allowed urls (no need for direct declaration)

Installation goes currently via:
curl -s https://raw.githubusercontent.com/hastmu/apt-proxy-detect/main/install.deb.sh | sudo bash