Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix endless 5xx responses leading to pages
The root cause here is that rails was returning responses to the reverse proxy where the total size of all the headers was greater than 4k. This would lead to an `upstream sent too big header while reading response header from upstream` error in nginx, resulting in a default 502 bad gateway response. It took us dozens of hours of searching to find the root cause because our nginx log parsing was throwing away all of the error logs, and I only saw it when directly tailing the logs from the container in k8s. As to how the rails app was returning so many header bytes: The `Redirector` middleware generates a 301 whenever it sees a host that does not match a predefined list. Notably, `api.rubygems.org` points to prod, but is _not_ in that list. The 301 includes a `Location` header with the correct host, while maintaining the rest of the path and query string. This means that the `Location` header could contain up to the nginx limit of almost 8k bytes. In combination with the other headers returned by the rails app, this was sufficient to exceed the default limit of 4k bytes. Signed-off-by: Samuel Giddins <segiddins@segiddins.me>
- Loading branch information