azureredisstackexchange.redis

StackExchange.Redis.RedisTimeoutException Timeout performing EXISTS (Since moving WebApp to .NET8)


I'm sure the issue here is not with StackExchange.Redis but something in our Azure App Service config/startup/setup.

We have been using StackExchange.Redis happily for years. We migrated an Azure App Service to .NET 8, and now redis is very slow and frequently errors:

Timeout performing EXISTS (5000ms), 
next: EXISTS XXX, 
inst: 10, qu: 0, qs: 0, 
aw: False, bw: SpinningDown, 
rs: ReadAsync, 
ws: Idle, in: 208, 
last-in: 0, cur-in: 0, sync-ops: 20932, 
async-ops: 1, 
serverEndpoint: XXX-XXXXX-REDIS-ECOM-PRD-1-STAGE.redis.cache.windows.net:6380, 
conn-sec: 64302.22, aoc: 0, mc: 1/1/0, 
mgr: 10 of 10 available, 
clientName: DW0MDWK0000LG(SE.Redis-v2.8.0.27420), 
IOCP: (Busy=0,Free=1000,Min=1,Max=1000), 
WORKER: (Busy=37,Free=32730,Min=2,Max=32767), 
POOL: (Threads=37,QueuedItems=64,CompletedItems=315273,Timers=29), 
v: 2.8.0.27420 
Failed Method: StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl

My gut feeling is this has to do with the resources of the Azure App in some way.

I could increase the timeout, but redis should be returning these light queries in a lot less than 5000ms. In the Az portal, the Redis Server Load is under 10%. We hardly use it. Mem is 40Mb of 250Mb.

 slowlog get 10
 1) 1) (integer) 260
    2) (integer) 1725447601
    3) (integer) 30987
    4) 1) "EVALSHA"
       2) "3915ee22fda531a1d5661f2523d0443fd35ff0a4"
       3) "1"
       4) "CatalogueDataAccess.GetX"
       5) "638610516012637021"
       6) "-1"
       7) "7200"
       8) A small amount of JSON here

I have tried and failed to interpret the error messages.

Can a Redis wizard please interpret.


Solution

  • POOL: (Threads=37,QueuedItems=64,CompletedItems=315273,Timers=29),
    

    Low threads + high QueuedItems + low CPU usage indicates thread pool exhaustion. Happened to our app on .net 8 upgrade too.

    StackExchange.Redis still relies on shared thread pool, and with a high request rate it just fails first, so you see RedisTimeout exceptions. However, anything that relies on thread poll, including asp.net, will be slow because of exhaustion as well. Redis client is just a canary that dies first.

    The fix is to set min threads to 200-300 to allow .net to spin up the pool quickly.
    Add the following on the application start:

    ThreadPool.SetMinThreads(300, 300);
    

    Ref: Azure Cache for Redis management FAQs / Important details about ThreadPool growth