-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Socket hangup error #299
Comments
I noticed that the issue only seems to show up when I run all the smoke tests at once. if I use for example It's not clear to me that this is happening only for large datasources. For example the kelp regional rasters aren't particularly big and I'm getting the socket hang up error for thos. This issue with node-fetch seems to match the issue but I've already upgraded to the latest node-fetch version that should have the fix -- node-fetch/node-fetch#1735. I'm curious @avmey, do you see the errors in deployment (I'm guessing you mean you see it in the AWS logs) at the same frequency and for the same datasources as in the smoke tests? |
node-fetch fixed its bug in 2.6.13 as well as 3.3.2. I'm noticing that for the california project at least all of the socket hang ups except one are raster files. I wonder if it's an upstream issue. Digging into the package-lock.json, it looks like geoblaze uses an older version of cross-fetch which uses node-fetch 2.6.12. I'm not sure this is the root cause but if so I'm not sure cross-fetch is actively maintained to fix it, because Node 22 removing the need for node-fetch and cross-fetch altogether but Lambda can't use Node v22 until November 2024 - lquixada/cross-fetch#183. |
While cross-fetch may still suggest node-fetch |
Here's a network that in deployment is experiencing the error. (The error is caught and displayed in the reports). On my end, kelp forests (raster) and shoreline habitats (vector) reports are both experiencing it, sometimes rock islands (vector) report as well. It's frequent - when I run a network there's always a couple errored reports. |
It looks like Node 19 and above by default now has https://connectreport.com/blog/tuning-http-keep-alive-in-node-js/ I'm going to start with trying to set keepalive to false, to see if that addresses it. And we can revisit how to add it back. I'm not sure that geoblaze provides access to passing fetch parameters but I can start with the flatgeobuf client. Can then look at bringing back keepalive, if it can be configured properly. Node also possibly has maxSockets set to infinite by default. |
Some additional info on configuring keepalive like timeout - https://betterstack.com/community/guides/scaling-nodejs/nodejs-errors/#2-econnreset |
In both testing and deployment, since upgrading to the beta 7.0.x, I've starting seeing socket hangup errors with larger datasets. Seems to affect both rasters and vectors.
In test:
In deployment:
The text was updated successfully, but these errors were encountered: