Too many open files

[bchartier]

On a newly installed SDI-CC instance on a Linux desktop computer, we encountered this error whiling runing the checks on DataGrandEst full WMS service: https://www.datagrandest.fr/geoserver/ows

Unable to parse the remote OWS server: HTTPSConnectionPool(host='www.datagrandest.fr', port=443): 
Max retries exceeded with url: 
/geoserver/ows?SERVICE=WMS&&service=WMS&version=1.3.0&request=GetMap&layers=lbwgs1%3ALBWGS1_M_640_Nr_11997&styles=&width=10&height=10&crs=EPSG%3A4326&bbox=48.04533868591011%2C7.080232317672589%2C48.048442572946854%2C7.084768200056035&format=image%2Fpng&transparent=FALSE&exceptions=XML&bgcolor=0xFFFFFF
(Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f24bfd66f00>: 
Failed to establish a new connection: [Errno 24] Too many open files'))

This kind of HTTP requests are managed with the requests library (encapsulated within owslib code) without using a requests Session object. Therefore for each URL tested by SDI-CC a new socket is opened. Using the requests Session class would avoid this kind of error and would improve performance.

Docs about the Session class : https://docs.python-requests.org/en/latest/user/advanced/#session-objects

[zulf10]

To reproduce, launch python3 checker.py --mode WMS --server https://www.datagrandest.fr/geoserver/ows\?service\=wms\&version\=1.1.1\&request\=GetCapabilities --log-to-file output.log

• Check python pid ps aux | grep python
• List files open with pid ls -al /proc/792173/fd/
• Check the number of files from time to time ls -al /proc/792173/fd/ | wc -l and TCP connections ss -s
  Each connection still open socket (with file)

I'm on Fedora workstation on my Laptop. Linux basic limit is 1024 open files.
ulimit -a show this limit

To increase the limit : ulimit -n 4096
and re-launch python3 checker.py

Today, the software opens the 2133 layers
So 2133 sockets and files

edit: I don't think it's a good idea not to reuse one or more open sockets