Allow to set registry mirror

[mokazemi]

Hi. The server that I'm using for Nextcloud AIO is located in Iran and have some problems in accessing to auth.docker.io. I usually set a registry mirror for my docker like this:

{
  "registry-mirrors": ["https://registry.docker.ir"]
}

But the issue is, the nextcloud/all-in-one internally tires to access to the docker.io to check for updates. Now I can't even open the Nextcloud AIO Interface page, and I see some errors on the logs of the container:

[Sun Sep 15 08:27:09.307602 2024] [core:notice] [pid 188:tid 188] AH00094: Command line: 'httpd -D FOREGROUND'
NOTICE: PHP message: Could not get digest of container nextcloud/all-in-one:latest cURL error 28: Failed to connect to auth.docker.io port 443 after 300002 ms: Timeout was reached (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://auth.docker.io/token?service=registry.docker.io&scope=repository:nextcloud/all-in-one:pull
NOTICE: PHP message: Could not get digest of container nextcloud/aio-apache:latest cURL error 28: Failed to connect to auth.docker.io port 443 after 300000 ms: Timeout was reached (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://auth.docker.io/token?service=registry.docker.io&scope=repository:nextcloud/aio-apache:pull

How should I solve the issue? Could there be an option to use system registry mirrors instead of using the default one or a env var for it? Or anything else I can do?

[szaimen]

Hi, thanks for your report. In general we might be able to retrieve this value via docker info. However I would need to know where to reach the authentication domain as this is apparently the one that is not reachable:

Timeout was reached (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://auth.docker.io

[szaimen]

Is there maybe also a mirror for that?

[mokazemi]

Oh, sorry I didn't see your reply.
Could you please explain more on this? What info should I provide?

[szaimen]

See #5268 (reply in thread)

[houming818]

The problem is the same as in .cn

[aojie654]

@szaimen Can we config it to self-buildup registry mirror in anywhere please?

[szaimen]

Can you please answer my question above?

[aojie654]

Hi Simon,

Is it possible to use a config file or variable to set the registry mirror for some domains? e.g.

[MIRRORS]
auth.docker.io = mirror-auth.example.com
hub.docker.com = mirror-hub.example.com

[szaimen]

Currently it is not but I was wondering if there is a value in docker itself where you can modify auth.docker.io and hub.docker.com in the daemon.json of docker that we could retreive maybe via sudo docker info?

[aojie654]

Looks there is too many issues from some region with network blocking, so I set up Nextcloud manually with php-8.3 and Nginx. 🤔

[invis-z]

I would need to know where to reach the authentication domain as this is apparently the one that is not reachable:

Timeout was reached (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://auth.docker.io/

Is there maybe also a mirror for that?

@szaimen Based on my understanding, the auth.docker.io domain is not in any part of the code, instead it comes from the registry server

$ curl -iL index.docker.io/v2
HTTP/1.1 301 Moved Permanently
content-length: 0
location: https://index.docker.io/v2

HTTP/1.1 301 Moved Permanently
content-type: text/html; charset=utf-8
docker-distribution-api-version: registry/2.0
location: /v2/
date: Thu, 16 Jan 2025 07:18:13 GMT
content-length: 39
strict-transport-security: max-age=31536000

HTTP/1.1 401 Unauthorized
content-type: application/json
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"
date: Thu, 16 Jan 2025 07:18:13 GMT
content-length: 87
strict-transport-security: max-age=31536000

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

In the last response there is a www-authenticate header and it gives the domain. In theory, the mirror will return its own domain and url that would be different for each server.

PS. I referenced watchtower's code to figure this out as I was also facing this problem with it. Link

[invis-z]

Quickly spun up a container for a local registry and here is what I got:

$ curl -iL localhost:5000/v2
HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Location: /v2/
Date: Thu, 16 Jan 2025 07:49:14 GMT
Content-Length: 39

HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Basic realm="Registry Realm"
X-Content-Type-Options: nosniff
Date: Thu, 16 Jan 2025 07:49:14 GMT
Content-Length: 87

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

In this case, there will not be another domain for the authentication and it will be done through http basic auth.

Here basic auth is what I set up, it might possible to do Bearer but the set up will be more complicated. In that case it will return a similar header as the official registry but with the server's own domain.

If the mirror didn't set up auth, it will just return a 200:

$ curl -iL https://my-local-mirror.example/v2
HTTP/2 301 
content-type: text/html; charset=utf-8
docker-distribution-api-version: registry/2.0
location: /v2/
content-length: 39
date: Thu, 16 Jan 2025 07:53:46 GMT

HTTP/2 200 
content-type: application/json; charset=utf-8
docker-distribution-api-version: registry/2.0
x-content-type-options: nosniff
content-length: 2
date: Thu, 16 Jan 2025 07:53:46 GMT

{}

[invis-z]

Also, by the way, containrrr/watchtower seems to be inactive. And the mirror will not work with it: containrrr/watchtower#1346. A fork or a switch to other projects might be needed.

[szaimen]

Hi, thanks for all the input. After thinking a bit more about this issue, the best solution from my POV is to add a check that blocks the installation if access to auth.docker.io is not possible: #5888

However it will also point everyone at https://github.com/nextcloud/all-in-one/tree/main/manual-install which allows to install the project in such environments.

[szaimen]

So what about auth.docker.io? Is this not needed in the case of a mirror?

[szaimen]

We need it in the normal installation to retreive a token so that we can check for image updates without downloading them

[invis-z]

The auth.docker.io should never been hardcoded, it should be queried from index.docker.io/v2 or registry-1.docker.io/v2 from the www-authenticate header of the response.

$ curl -iL registry-1.docker.io/v2
HTTP/1.1 301 Moved Permanently
content-length: 0
location: https://registry-1.docker.io/v2

HTTP/1.1 301 Moved Permanently
content-type: text/html; charset=utf-8
docker-distribution-api-version: registry/2.0
location: /v2/
date: Thu, 16 Jan 2025 10:51:12 GMT
content-length: 39
strict-transport-security: max-age=31536000

HTTP/1.1 401 Unauthorized
content-type: application/json
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io" <---------------------
date: Thu, 16 Jan 2025 10:51:12 GMT
content-length: 87
strict-transport-security: max-age=31536000

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

[invis-z]

In the case of a mirror, querying the mirror will return how it want to auth, most of the time this is not needed as docker actually doesn't handle mirror auth correctly. distribution/distribution#1486 (comment)
In this use case, since all the images are public accessible, if the mirror is set in daemon.json, it won't need to be authed so it will not need the token to access.

[szaimen]

I see. Thanks for the answers and explanation.

So this would need a bigger refactoring to make it work...

I fear I don't have the time to work on this, so PRs are welcome!