A solution to dynamically sized dict

[Kchour]

I was able to grow the underlying memory mapped file. In the dict.py module around line 184ish, I did the following:

def _save_memory(self, db: Dict[str, Any]) -> None:
        """Allocate more memory 1.5x current size
        if there is a ValueError
        
        """
        data = self._serializer.dumps(db)
        # get size of current memory block
        sz = len(self._memory_block.buf)
        while True:
            try:
                # keep trying to add additional data until success
                self._memory_block.buf[: len(data)] = data
                break
            except ValueError as exc:
                # raise ValueError("exceeds available storage") from exc

                # grow file size by 1.5 times
                new_size = int(1.5*sz)
                os.ftruncate(self.shm._fd, new_size) 
                stats = os.fstat(self.shm._fd)
                self.shm._size = stats.st_size
                self.shm._mmap = mmap.mmap(self.shm._fd, new_size)
                self.shm._buf = memoryview(self.shm._mmap)
            except Exception as exc:
                # Handle unexpected error
                raise exc

I wrote a simple test script:

from shared_memory_dict import SharedMemoryDict

# try to create a shared memory file of size 1024 bytes
# will reuse existing file if available
smd = SharedMemoryDict(name='tokens', size=1024)
smd['some-key'] = 'some-value-with-any-type'

print("File size before: ", smd.shm.size)

for i in range(400):
    smd[i] = i

print("File size after: ", smd.shm.size)

# clean up 
smd.shm.close()
smd.shm.unlink()

which ouputs:

File size before:  1024
File size after:  2304

Additionally I fired up a separate python process in a 2nd terminal window and was able to see all changes. This seems promising, but I haven't tested everything; this may not even be the best approach. Sometimes, the two processes appear "out of sync". This usually happens if you abruptly close a process before doing proper cleanup (.close() and .unlink()).

[wnark]

This is to create a new shared memory area and map it in？