how to control the max memory utilization that faiss search can take?

[n0thing233]

Hi Team,

I’m trying to load an index from disk in a memory-mapped way and perform a search on it. After loading the index using:
index = faiss.read_index('faiss.index', faiss.IO_FLAG_MMAP)
I noticed a slight increase in memory usage, which is expected. However, when I run a search:
distances, ids = index.search(query, k)
the memory usage spikes to 100%.

It seems that FAISS (via its C extension) aggressively utilizes memory during the search, especially if the index size exceeds available memory.

Is there a way to set a memory usage limit for FAISS during the search? My main challenge is running this code within a distributed framework like Dask. The Dask workers require their own memory allocation, but the FAISS search task consumes all available memory, causing the workers to fail.

Any advice or suggestions would be greatly appreciated. Thanks!

[bshethmeta]

How big is your index, query size and k & d?

I can imagine if you want to optimize for memory, you can split up your query into smaller chunks.

[n0thing233]

the index is 220GB+ , let's say the available memory is 60GB.
k = 1 , d = 512.
the query is like 2GB.
I thought faiss will do batching internally? in this case explicit smaller chunks won't help right?

I want if faiss or it's lower level dependencies (i.e openMP) have configuration to limit the memory it use?

[mdouze]

There are two possibilities here:

• the memory is used by the memory map, ie. the file data gets cached to RAM by the kernel. This is a normal behavior until the memory is exhausted. The memory will be freed when it is too tight.

• the search uses temporary memory. For example IVFPQ uses look-up tables allocated here: https://github.com/facebookresearch/faiss/blob/main/faiss/IndexIVFPQ.cpp#L518 but this depends on the type of index you use. There are also fixed precomputed tables computed here: http://github.com/facebookresearch/faiss/blob/main/faiss/IndexIVFPQ.cpp#L364
  but they should not grow beyond precomputed_table_max_bytes=2GiB by default.

So what type of index are you using?

[n0thing233]

I'm using IVF5000,Flat .
I think it falls into the first possibility where the memory is exhausted by memory map.
I wonder if there is any way to set a memory limit to the memory map and make it not greedily exhaust the memory.

[mdouze]

Ah right. This is an OS mem management problem. Probably some option to ulimit should be able to limit it (-v or -d), see https://linuxcommand.org/lc3_man_pages/ulimith.html

However, I would argue that using available memory to cache mmapped files is sane since this will take only unused memory.

[n0thing233]

Thanks, @mdouze,

In a scenario where FAISS search is run within a distributed processing framework like Dask, a Dask worker might initiate the FAISS search but later require additional memory for its own tasks, such as communicating with other workers. However, all available memory may already be consumed by FAISS mmap files.

it sounds like there is no way that we can set a memory limit to faiss search at application-level.
Unfortunately the dask worker and the faiss are in the same process, so ulimit won't work here.