1. Memory Backends and Allocators

Memory Backend

A memory backend represents a contiguous array of bytes which was allocated in bulk from a memory allocator provided by the Operating System (OS). Typically, a Memory Backend allocates a large amount of virtual memory since memory is not truly allocated until it is first initialized. The only limitation is the size of the virtual address space (typically 2^64 bytes).

class MemoryBackend {
public:
  MemoryBackendHeader *header_;
  char *data_;
  size_t data_size_;
  bitfield32_t flags_;
}

We currently provide the following Memory Backends:

1. Mmap (private): uses the mmap() system call to allocate private, anonymous memory
2. Mmap (shared): uses the mmap() system call to allocate shared memory
3. Array: The user inputs an already-allocated buffer. Does not internally allocate any memory.

Memory Allocator

A memory allocator manages an array of contiguous data, typically provided by a Memory Backend. Memory allocators fundamentally provide two interfaces: allocate and free. SHM allocators return offsets from a memory backend, instead of raw pointers.

class Allocator {
  virtual OffsetPointer Allocate(size_t size);
  virtual void Free(OffsetPointer &p);
}

There are 4 different pointer offset types:

1. OffsetPointer: stores a 64-bit offset
2. Pointer: stores a 64-bit offset + allocator ID (64-bit)
3. AtomicOffsetPointer: stores a 64-bit offset using atomic instructions (guarantees memory coherency)
4. AtomicPointer: stores the 64-bit offset using atomic instructions, but stores the allocator ID regularly

Creating a Backend and Allocator

The following example is in example/allocator.cc

#include <mpi.h>
#include <cassert>
#include "hermes_shm/data_structures/thread_unsafe/list.h"

struct CustomHeader {
  int data_;
};

int main(int argc, char **argv) {
  int rank;
  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  // Common allocator information
  std::string shm_url = "test_allocators";
  hipc::allocator_id_t alloc_id(0, 1);
  auto mem_mngr = HERMES_MEMORY_MANAGER;
  hipc::Allocator *alloc;
  CustomHeader *header;

  // Create backend + allocator
  if (rank == 0) {
    // Create a 64 megabyte allocatable region
    mem_mngr->CreateBackend<hipc::PosixShmMmap>(
      MEGABYTES(64), shm_url);
    // Create a memory allocator over the 64MB region
    alloc = mem_mngr->CreateAllocator<hipc::StackAllocator>(
      shm_url, alloc_id, sizeof(CustomHeader));
    // Get the custom header from the allocator
    header = alloc->GetCustomHeader<CustomHeader>();
    // Set custom header to 10
    header->data_ == 10;
  }
  MPI_Barrier(MPI_COMM_WORLD);

  // Attach backend + find allocator
  if (rank != 0) {
    mem_mngr->AttachBackend(hipc::MemoryBackendType::kPosixShmMmap, shm_url);
    alloc = mem_mngr->GetAllocator(alloc_id);
    header = alloc->GetCustomHeader<CustomHeader>();
  }
  MPI_Barrier(MPI_COMM_WORLD);

  // Verify header is equal to 10 in all processes
  assert(header->data_ == 10);

  // Finalize
  if (rank == 0) {
    std::cout << "COMPLETE!" << std::endl;
  }
  MPI_Finalize();
}