Memory used but not tracked by memray

[JazJaz426]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Hi,

I used memray to track the memory usage in the populate_index function and in the end of populate_index function there's another _make_vector_index function. My goal is to optimize the memory usage of this function as much as possible.

with memray.Tracker(output_path):
                self._text_embedding_vector_table.populate_index(
                    vector_store_dataset.schema.item.id,
                    text_embedding_transform,
                    inner_table=pa.Table.from_pandas(vector_store_dataset.items),
                )

def populate_index():
    .....
    self._make_vector_index(index_dims)

Initially before optimization, as shown in the graph below there's no make_vector_index function at all and I tried to optimize for [feature_matrix.data[col].to_list() for col in vector_columns], which takes ~150MB out of 300MB memory populate_index consumes.

After optimization, [feature_matrix.data[col].to_list() for col in vector_columns], is replaced by more efficient implementation and it disappears from the new flamegraph. However, the _make_vector_index appears, which takes 150MB out of ~270MB total memory populate_index consumes. Even oddly, when I profile _make_vector_index individually before and after optimization, it turns out it always consume the same memory (150MB) even though it didn't show up in the previous flamegraph at all. Note that the optimization for [feature_matrix.data[col].to_list() for col in vector_columns], is not related to make_vector_index.

So super confused why the flamegraph hided the _make_vector_index memory usage even it consumes signifcant memory?

Expected Behavior

No response

Steps To Reproduce

Not sure how to reproduce

Memray Version

1.15.0

Python Version

3.9

Operating System

macOS

Anything else?

No response

[pablogsal]

Hi 👋

Thanks for opening an issue. Unfortunately without a reproducer we will not be able to help a lot here since we cannot check anything ourselves and investigate.

Could you please run again without pymalloc by either setting the PYTHONMALLOC=malloc or by using the --trace-python-allocators flag?

[pablogsal]

I suspect it may be the allocator layer. Take a look at https://bloomberg.github.io/memray/python_allocators.html

[pablogsal]

I converted this to a discussion because I am not sure this is a bug.