Skip to content

Commit

Permalink
Fix endless 5xx responses leading to pages
Browse files Browse the repository at this point in the history
The root cause here is that rails was returning
responses to the reverse proxy where the total
size of all the headers was greater than 4k. This
would lead to an
`upstream sent too big header while reading response header from upstream`
error in nginx, resulting in a default 502 bad
gateway response.

It took us dozens of hours of searching to find
the root cause because our nginx log parsing was
throwing away all of the error logs, and I only
saw it when directly tailing the logs from the
container in k8s.

As to how the rails app was returning so many
header bytes:
The `Redirector` middleware generates a 301
whenever it sees a host that does not match a
predefined list. Notably, `api.rubygems.org`
points to prod, but is _not_ in that list.
The 301 includes a `Location` header with the
correct host, while maintaining the rest of the
path and query string. This means that the `Location`
header could contain up to the nginx limit of almost
8k bytes. In combination with the other headers returned
by the rails app, this was sufficient to exceed the
default limit of 4k bytes.

Signed-off-by: Samuel Giddins <segiddins@segiddins.me>
  • Loading branch information
segiddins committed Jan 14, 2025
1 parent cecf79b commit 10cd26c
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions config/deploy/nginx-configmap.yaml.erb
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ data:
worker_connections 5120;
}
http {
# Ensure buffer is large enough to contend with a redirect to a URL that is at the request line length limit
proxy_buffers 4 16k;
proxy_buffer_size 16k;

server {
listen 80;
server_name _;
Expand Down

0 comments on commit 10cd26c

Please sign in to comment.