[GH Request] CI checks failing due to Dockerhub rate limiting

[bmtcril]

Firm Name

Axim

Urgency

Medium (< 2 weeks)

Problem/Request

From https://axim-collaborative.slack.com/archives/C12M8M5AR/p1736434755199059 :

Yesterday and today I've seen edx-platform CI checks failing intermittently with this error:
Unable to find image 'mongo:7.0' locally
7.0: Pulling from library/mongo
docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
Is this familiar to anyone? Wondering if we need to change something on the runners.

We should see if we can do something to cache / mirror our image pulls in CI, which may speed things up in addition to reducing our pull rates.

Reasoning

We'd like CI to pass reliably. :sli

[github-actions]

Thank you for your report! @openedx/axim-oncall will triage within a business day. Simple requests usually take 2-3 business days to resolve; more complex requests could take longer.

[bmtcril]

openedx/edx-platform#36089 is our best guess right now. Dockerhub seemed to be rate limiting us based on an unauthenticated threshold, so we're authenticating before the pull now. It doesn't seem to harm anything, if it lands I can monitor for those failures to see if it solves things.

[bmtcril]

This PR broke fork CI, and was subsequently fixed in openedx/edx-platform#36100 . That seems to be have solved or at least lessened the issue, so I'm closing this for now. I'll continue monitoring for the next few days.

[timmc-edx]

Looks like this might not have fixed it... https://github.com/openedx/edx-platform/actions/runs/12775202089/job/35611041323

[robrap]

[inform] I opened a newer ticket, not know this existed. Feel free to work from whichever ticket makes more sense. See:

• [GH Request] Flaky edx-platform builds due to docker rate limit #1357

[bmtcril]

Thanks for the heads up, I'll reopen this one and work from here.

[bmtcril]

openedx/edx-platform#36110 is the latest. The caching works, but it takes a full run to take effect since all of the jobs kick off at the same time and the cache isn't written until the job completes.

[bmtcril]

I haven't seen any more of these since the last changes, so I'm going to close again. There are some other possible places where we may get some small benefit from caching, but mostly they're using service entries and I'm not at all certain it will help there.

[robrap]

Thank you and 🤞.