Skip to content

Commit

Permalink
Expose 'eviction' and 'moving' states for an item
Browse files Browse the repository at this point in the history 
markForEviction is used only in findEviction and
evictForSlabRelease but not for item movement.

moveForSlabRelease relies on markMoving().

Only allow to mark item for eviction if ref count
is 0. This ensures that after item is marked, eviction
cannot fail. This makes it possible to return NULL handle
immediately from find if item is marked for eviction.

markMoving() does have those restrictions and still allows
readers to obtain a handle to a moving item.

Also, add option to use combined locking for MMContainer
iteration.

Pass item ref to NavyCache::put
  • Loading branch information
@igchor
igchor committed Dec 8, 2022
1 parent 94eaded commit e60eeed
Show file tree
Hide file tree
Showing 27 changed files with 762 additions and 876 deletions.
650 changes: 251 additions & 399 deletions cachelib/allocator/CacheAllocator-inl.h
View file

Large diffs are not rendered by default.

67 changes: 20 additions & 47 deletions cachelib/allocator/CacheAllocator.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1308,7 +1308,7 @@ class CacheAllocator : public CacheBase {

private:
// wrapper around Item's refcount and active handle tracking
FOLLY_ALWAYS_INLINE void incRef(Item& it);
FOLLY_ALWAYS_INLINE bool incRef(Item& it);
FOLLY_ALWAYS_INLINE RefcountWithFlags::Value decRef(Item& it);

// drops the refcount and if needed, frees the allocation back to the memory
Expand Down Expand Up @@ -1359,6 +1359,12 @@ class CacheAllocator : public CacheBase {
bool nascent = false,
const Item* toRecycle = nullptr);

// Must be called by the thread which called markForEviction and
// succeeded. After this call, the item is unlinked from Access and
// MM Containers. The item is no longer marked as exclusive and it's
// ref count is 0 - it's available for recycling.
void unlinkItemForEviction(Item& it);

// acquires an handle on the item. returns an empty handle if it is null.
// @param it pointer to an item
// @return WriteHandle return a handle to this item
Expand Down Expand Up @@ -1448,17 +1454,17 @@ class CacheAllocator : public CacheBase {
// @return handle to the parent item if the validations pass
// otherwise, an empty Handle is returned.
//
ReadHandle validateAndGetParentHandleForChainedMoveLocked(
WriteHandle validateAndGetParentHandleForChainedMoveLocked(
const ChainedItem& item, const Key& parentKey);

// Given an existing item, allocate a new one for the
// existing one to later be moved into.
//
// @param oldItem the item we want to allocate a new item for
// @param item reference to the item we want to allocate a new item for
//
// @return handle to the newly allocated item
//
WriteHandle allocateNewItemForOldItem(const Item& oldItem);
WriteHandle allocateNewItemForOldItem(const Item& item);

// internal helper that grabs a refcounted handle to the item. This does
// not record the access to reflect in the mmContainer.
Expand Down Expand Up @@ -1512,7 +1518,7 @@ class CacheAllocator : public CacheBase {
// callback is responsible for copying the contents and fixing the semantics
// of chained item.
//
// @param oldItem Reference to the item being moved
// @param oldItem item being moved
// @param newItemHdl Reference to the handle of the new item being moved into
//
// @return true If the move was completed, and the containers were updated
Expand Down Expand Up @@ -1662,25 +1668,6 @@ class CacheAllocator : public CacheBase {

using EvictionIterator = typename MMContainer::Iterator;

// Advance the current iterator and try to evict a regular item
//
// @param mmContainer the container to look for evictions.
// @param itr iterator holding the item
//
// @return valid handle to regular item on success. This will be the last
// handle to the item. On failure an empty handle.
WriteHandle advanceIteratorAndTryEvictRegularItem(MMContainer& mmContainer,
EvictionIterator& itr);

// Advance the current iterator and try to evict a chained item
// Iterator may also be reset during the course of this function
//
// @param itr iterator holding the item
//
// @return valid handle to the parent item on success. This will be the last
// handle to the item
WriteHandle advanceIteratorAndTryEvictChainedItem(EvictionIterator& itr);

// Deserializer CacheAllocatorMetadata and verify the version
//
// @param deserializer Deserializer object
Expand Down Expand Up @@ -1756,22 +1743,23 @@ class CacheAllocator : public CacheBase {

// @return true when successfully marked as moving,
// fasle when this item has already been freed
bool markExclusiveForSlabRelease(const SlabReleaseContext& ctx,
void* alloc,
util::Throttler& throttler);
bool markMovingForSlabRelease(const SlabReleaseContext& ctx,
void* alloc,
util::Throttler& throttler);

// "Move" (by copying) the content in this item to another memory
// location by invoking the move callback.
//
//
// @param ctx slab release context
// @param item old item to be moved elsewhere
// @param oldItem old item to be moved elsewhere
// @param handle handle to the item or to it's parent (if chained)
// @param throttler slow this function down as not to take too much cpu
//
// @return true if the item has been moved
// false if we have exhausted moving attempts
bool moveForSlabRelease(const SlabReleaseContext& ctx,
Item& item,
Item& oldItem,
util::Throttler& throttler);

// "Move" (by copying) the content in this item to another memory
Expand All @@ -1794,18 +1782,7 @@ class CacheAllocator : public CacheBase {
Item& item,
util::Throttler& throttler);

// Helper function to evict a normal item for slab release
//
// @return last handle for corresponding to item on success. empty handle on
// failure. caller can retry if needed.
WriteHandle evictNormalItemForSlabRelease(Item& item);

// Helper function to evict a child item for slab release
// As a side effect, the parent item is also evicted
//
// @return last handle to the parent item of the child on success. empty
// handle on failure. caller can retry.
WriteHandle evictChainedItemForSlabRelease(ChainedItem& item);
typename NvmCacheT::PutToken createPutToken(Item& item);

// Helper function to remove a item if expired.
//
Expand Down Expand Up @@ -1927,18 +1904,14 @@ class CacheAllocator : public CacheBase {
std::optional<bool> saveNvmCache();
void saveRamCache();

static bool itemExclusivePredicate(const Item& item) {
return item.getRefCount() == 0;
static bool itemSlabMovePredicate(const Item& item) {
return item.isMoving() && item.getRefCount() == 0;
}

static bool itemExpiryPredicate(const Item& item) {
return item.getRefCount() == 1 && item.isExpired();
}

static bool parentEvictForSlabReleasePredicate(const Item& item) {
return item.getRefCount() == 1 && !item.isExclusive();
}

std::unique_ptr<Deserializer> createDeserializer();

// Execute func on each item. `func` can throw exception but must ensure
Expand Down
52 changes: 37 additions & 15 deletions cachelib/allocator/CacheItem-inl.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -148,13 +148,14 @@ std::string CacheItem<CacheTrait>::toString() const {
return folly::sformat(
"item: "
"memory={}:raw-ref={}:size={}:key={}:hex-key={}:"
"isInMMContainer={}:isAccessible={}:isExclusive={}:references={}:ctime="
"isInMMContainer={}:isAccessible={}:isMarkedForEviction={}:references={"
"}:ctime="
"{}:"
"expTime={}:updateTime={}:isNvmClean={}:isNvmEvicted={}:hasChainedItem="
"{}",
this, getRefCountAndFlagsRaw(), getSize(),
folly::humanify(getKey().str()), folly::hexlify(getKey()),
isInMMContainer(), isAccessible(), isExclusive(), getRefCount(),
isInMMContainer(), isAccessible(), isMarkedForEviction(), getRefCount(),
getCreationTime(), getExpiryTime(), getLastAccessTime(), isNvmClean(),
isNvmEvicted(), hasChainedItem());
}
Expand Down Expand Up @@ -217,23 +218,43 @@ bool CacheItem<CacheTrait>::isInMMContainer() const noexcept {
}

template <typename CacheTrait>
bool CacheItem<CacheTrait>::markExclusive() noexcept {
return ref_.markExclusive();
bool CacheItem<CacheTrait>::markForEviction() noexcept {
return ref_.markForEviction();
}

template <typename CacheTrait>
RefcountWithFlags::Value CacheItem<CacheTrait>::unmarkExclusive() noexcept {
return ref_.unmarkExclusive();
RefcountWithFlags::Value CacheItem<CacheTrait>::unmarkForEviction() noexcept {
return ref_.unmarkForEviction();
}

template <typename CacheTrait>
bool CacheItem<CacheTrait>::isExclusive() const noexcept {
return ref_.isExclusive();
bool CacheItem<CacheTrait>::isMarkedForEviction() const noexcept {
return ref_.isMarkedForEviction();
}

template <typename CacheTrait>
bool CacheItem<CacheTrait>::isOnlyExclusive() const noexcept {
return ref_.isOnlyExclusive();
bool CacheItem<CacheTrait>::markForEvictionWhenMoving() {
return ref_.markForEvictionWhenMoving();
}

template <typename CacheTrait>
bool CacheItem<CacheTrait>::markMoving() {
return ref_.markMoving();
}

template <typename CacheTrait>
RefcountWithFlags::Value CacheItem<CacheTrait>::unmarkMoving() noexcept {
return ref_.unmarkMoving();
}

template <typename CacheTrait>
bool CacheItem<CacheTrait>::isMoving() const noexcept {
return ref_.isMoving();
}

template <typename CacheTrait>
bool CacheItem<CacheTrait>::isOnlyMoving() const noexcept {
return ref_.isOnlyMoving();
}

template <typename CacheTrait>
Expand Down Expand Up @@ -335,7 +356,7 @@ bool CacheItem<CacheTrait>::updateExpiryTime(uint32_t expiryTimeSecs) noexcept {
// check for moving to make sure we are not updating the expiry time while at
// the same time re-allocating the item with the old state of the expiry time
// in moveRegularItem(). See D6852328
if (isExclusive() || !isInMMContainer() || isChainedItem()) {
if (isMarkedForEviction() || !isInMMContainer() || isChainedItem()) {
return false;
}
// attempt to atomically update the value of expiryTime
Expand Down Expand Up @@ -451,13 +472,14 @@ std::string CacheChainedItem<CacheTrait>::toString() const {
return folly::sformat(
"chained item: "
"memory={}:raw-ref={}:size={}:parent-compressed-ptr={}:"
"isInMMContainer={}:isAccessible={}:isExclusive={}:references={}:ctime={}"
"isInMMContainer={}:isAccessible={}:isMarkedForEviction={}:references={}:"
"ctime={}"
":"
"expTime={}:updateTime={}",
this, Item::getRefCountAndFlagsRaw(), Item::getSize(), cPtr.getRaw(),
Item::isInMMContainer(), Item::isAccessible(), Item::isExclusive(),
Item::getRefCount(), Item::getCreationTime(), Item::getExpiryTime(),
Item::getLastAccessTime());
Item::isInMMContainer(), Item::isAccessible(),
Item::isMarkedForEviction(), Item::getRefCount(), Item::getCreationTime(),
Item::getExpiryTime(), Item::getLastAccessTime());
}

template <typename CacheTrait>
Expand Down
57 changes: 41 additions & 16 deletions cachelib/allocator/CacheItem.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -305,12 +305,17 @@ class CACHELIB_PACKED_ATTR CacheItem {
*/
RefcountWithFlags::Value getRefCountAndFlagsRaw() const noexcept;

FOLLY_ALWAYS_INLINE void incRef() {
if (LIKELY(ref_.incRef())) {
return;
// Increments item's ref count
//
// @return true on success, failure if item is marked as exclusive
// @throw exception::RefcountOverflow on ref count overflow
FOLLY_ALWAYS_INLINE bool incRef() {
try {
return ref_.incRef();
} catch (exception::RefcountOverflow& e) {
throw exception::RefcountOverflow(
folly::sformat("{} item: {}", e.what(), toString()));
}
throw exception::RefcountOverflow(
folly::sformat("Refcount maxed out. item: {}", toString()));
}

FOLLY_ALWAYS_INLINE RefcountWithFlags::Value decRef() {
Expand Down Expand Up @@ -344,23 +349,43 @@ class CACHELIB_PACKED_ATTR CacheItem {

/**
* The following two functions corresond to whether or not an item is
* currently in the process of being moved. This happens during a slab
* rebalance, eviction or resize operation.
* currently in the process of being evicted.
*
* An item can only be marked exclusive when `isInMMContainer` returns true.
* An item can only be marked exclusive when `isInMMContainer` returns true
* and item is not already exclusive nor moving and the ref count is 0.
* This operation is atomic.
*
* User can also query if an item "isOnlyExclusive". This returns true only
* if the refcount is 0 and only the exclusive bit is set.
*
* Unmarking exclusive does not depend on `isInMMContainer`.
* Unmarking exclusive does not depend on `isInMMContainer`
* Unmarking exclusive will also return the refcount at the moment of
* unmarking.
*/
bool markExclusive() noexcept;
RefcountWithFlags::Value unmarkExclusive() noexcept;
bool isExclusive() const noexcept;
bool isOnlyExclusive() const noexcept;
bool markForEviction() noexcept;
RefcountWithFlags::Value unmarkForEviction() noexcept;
bool isMarkedForEviction() const noexcept;

/**
* The following functions correspond to whether or not an item is
* currently in the processed of being moved. When moving, ref count
* is always >= 1.
*
* An item can only be marked moving when `isInMMContainer` returns true
* and item is not already exclusive nor moving.
*
* User can also query if an item "isOnlyMoving". This returns true only
* if the refcount is one and only the exclusive bit is set.
*
* Unmarking moving does not depend on `isInMMContainer`
* Unmarking moving will also return the refcount at the moment of
* unmarking.
*/
bool markMoving();
RefcountWithFlags::Value unmarkMoving() noexcept;
bool isMoving() const noexcept;
bool isOnlyMoving() const noexcept;

/** This function attempts to mark item as exclusive.
* Can only be called on the item that is moving.*/
bool markForEvictionWhenMoving();

/**
* Item cannot be marked both chained allocation and
Expand Down
31 changes: 3 additions & 28 deletions cachelib/allocator/MM2Q-inl.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -241,29 +241,9 @@ bool MM2Q::Container<T, HookPtr>::add(T& node) noexcept {
}

template <typename T, MM2Q::Hook<T> T::*HookPtr>
typename MM2Q::Container<T, HookPtr>::Iterator
MM2Q::Container<T, HookPtr>::getEvictionIterator() const noexcept {
// we cannot use combined critical sections with folly::DistributedMutex here
// because the lock is held for the lifetime of the eviction iterator. In
// other words, the abstraction of the iterator just does not lend itself well
// to combinable critical sections as the user can hold the lock for an
// arbitrary amount of time outside a lambda-friendly piece of code (eg. they
// can return the iterator from functions, pass it to functions, etc)
//
// it would be theoretically possible to refactor this interface into
// something like the following to allow combining
//
// mm2q.withEvictionIterator([&](auto iterator) {
// // user code
// });
//
// at the time of writing it is unclear if the gains from combining are
// reasonable justification for the codemod required to achieve combinability
// as we don't expect this critical section to be the hotspot in user code.
// This is however subject to change at some time in the future as and when
// this assertion becomes false.
LockHolder l(*lruMutex_);
return Iterator{std::move(l), lru_.rbegin()};
template <typename F>
void MM2Q::Container<T, HookPtr>::withEvictionIterator(F&& fun) {
lruMutex_->lock_combine([this, &fun]() { fun(Iterator{lru_.rbegin()}); });
}

template <typename T, MM2Q::Hook<T> T::*HookPtr>
Expand Down Expand Up @@ -460,10 +440,5 @@ void MM2Q::Container<T, HookPtr>::reconfigureLocked(const Time& currTime) {
lruRefreshTime_.store(lruRefreshTime, std::memory_order_relaxed);
}

// Iterator Context Implementation
template <typename T, MM2Q::Hook<T> T::*HookPtr>
MM2Q::Container<T, HookPtr>::Iterator::Iterator(
LockHolder l, const typename LruList::Iterator& iter) noexcept
: LruList::Iterator(iter), l_(std::move(l)) {}
} // namespace cachelib
} // namespace facebook
Loading

0 comments on commit e60eeed

Please sign in to comment.