Look into memory leaks

[tepickering]

This is related to MMTObservatory/WFSsrv#4, but seems to be even more of an issue with these servers. There is really no apparent reason for these processes to use up 100's of MB of RAM when they're not being actively used. First step will be to go through and make sure all callbacks exit cleanly using self.finish(). Then go through and manually delete things once they're not needed and maybe add some calls to gc to force garbage collection to happen.

[tepickering]

This is a pretty serious issue. After a week or so of uptime, the f9wfs and matcam servers were using over 2 GB of memory. And the matcam server was hardly used, if at all:

[tim@hacksaw ~]$ ps auwx | grep conda
mmtop     2678  6.3 24.2 2514744 1984336 ?     Ssl  Nov21 563:27 /mmt/conda/bin/python /mmt/camsrv/scripts/matcam.py
mmtop     2959  3.1 10.1 1716792 829464 ?      Ssl  Nov21 276:25 /mmt/conda/bin/python /mmt/WFSsrv/wfssrv/wfssrv.py
tim       3645  0.0  0.0 119468   892 pts/0    S+   08:42   0:00 grep --color=auto conda
mmtop     3691  0.0  0.0 151320  7320 ?        Ssl  07:36   0:00 tclsh /mmt/conda/bin/sao-wfs-server
mmtop    18944  6.7 28.4 2747828 2326120 ?     Rsl  Nov22 508:20 /mmt/conda/bin/python /mmt/camsrv/scripts/f9wfs.py

The wfssrv process is a lot more stable while also being much more heavily used. Will need to do some profiling of the servers that communicate with SBIG cameras to see what's using all of the memory.

[tepickering]

The core of the problem here lies within indiclient. The leakage has been somewhat mitigated, but the core issue of stale things not being garbage collected remains. May require a re-write of it to be fully async instead of threaded.