40x errors not counted against request limits

[eaton]

Need to do some additional testing, bit it appears that 404 errors and other failures allow unthrottled requests to continue hitting the scraped server. This includes 429 errors, which leads to the counterproductive outcome of rate limit errors triggering unthrottled crawling.

Not sure if this is an issue in the underlying Crawlee implementation of rate limiting, or a place where we're overriding some edge case behavior, but it should be investigated.