Preventing Cache Stampede in High-Traffic APIs with Double-Checked Locking

Preventing Cache Stampede in High-Traffic APIs with Double-Checked Locking

As the number of users of your website grows and more requests are sent to your server, the first component that comes under pressure is usually the database. To reduce the load, applications commonly introduce a caching layer in front of the database.

With this approach, the application first checks whether the requested data exists in the cache. If it does, the data is returned directly from the cache. Otherwise, the application retrieves the data from the database, stores it in the cache, and returns it to the client. Subsequent requests can then be served from the cache, significantly reducing the number of database queries.

But what happens when many requests arrive at exactly the same time?

Imagine that 10,000 concurrent requests are sent to the same API endpoint. If all of these requests reach the cache lookup at the same moment, they will all see that the cache is empty. As a result, every single request will query the database, making the cache ineffective in this scenario.

Consider the following code:

public async ValueTask<IReadOnlyList<LabelDto>> GetAllLabelsAsync(CancellationToken cancellationToken = default)
{
    if (_memoryCache.TryGetValue<List<LabelDto>>(AllLablesCacheKey, out var cachedItem) && cachedItem != null)
        return cachedItem;

    var labels = await _labelReadStore.GetAllLabelsAsync(cancellationToken);

    _memoryCache.Set(AllLablesCacheKey, labels, TimeSpan.FromMinutes(AbsoluteExpirationInMinutes));

    return labels;
}

If multiple requests reach the cache lookup simultaneously, they will all receive null. Each request will then query the database independently and store the same data in the cache. This results in thousands of unnecessary database queries.

So, how can we solve this problem?

The solution is to implement double-checked locking.

With this pattern, if 10,000 concurrent requests all find that the cache is empty, only one request is allowed to access the database. Once it retrieves the data, it stores the result in the cache. The remaining requests wait for the first request to complete and then read the data directly from the cache instead of querying the database themselves.

public async ValueTask<IReadOnlyList<LabelDto>> GetAllLabelsAsync(CancellationToken cancellationToken = default)
{
    if (_memoryCache.TryGetValue<List<LabelDto>>(AllLablesCacheKey, out var cachedItem) && cachedItem != null)
        return cachedItem;

    using (await AsyncLocker.LockAsync(AllLablesCacheKey))
    {
        if (_memoryCache.TryGetValue<List<LabelDto>>(AllLablesCacheKey, out cachedItem) && cachedItem != null)
            return cachedItem;

        var labels = await _labelReadStore.GetAllLabelsAsync(cancellationToken);

        _memoryCache.Set(AllLablesCacheKey, labels, TimeSpan.FromMinutes(AbsoluteExpirationInMinutes));

        return labels;
    }
}

This approach can have a significant impact on APIs that receive high volumes of concurrent requests, as it prevents duplicate database queries and protects the database from unnecessary load.

AsyncLocker Implementation:

public static class AsyncLocker
{
    private static readonly ConcurrentDictionary<string, SemaphoreSlim> _locks = new();

    public static async ValueTask<IDisposable> LockAsync(string key)
    {
        var semaphore = _locks.GetOrAdd(key, _ => new SemaphoreSlim(1, 1));

        await semaphore.WaitAsync();

        return new Releaser(key, semaphore);
    }

    private sealed class Releaser : IDisposable
    {
        private readonly string _key;
        private readonly SemaphoreSlim _semaphore;

        public Releaser(string key, SemaphoreSlim semaphore)
        {
            _key = key;
            _semaphore = semaphore;
        }

        public void Dispose()
        {
            _semaphore.Release();
        }
    }
}

Important Note:

The lock must be created per cache key, not globally. In other words, each cache key should have its own lock object. This ensures that only requests for the same cached resource wait for one another, while requests for different cache keys can continue executing concurrently.

This technique prevents unnecessary contention and allows your application to maintain high throughput while eliminating duplicate database queries for the same data.

Powered by Froala Editor