I'm implementing an RPC mechanism using redis.
A server connects to redis and listens for requests BLPOP job:queue 0, and a client connects and adds a request RPUSH job:queue [$job_id, ...]. The client then waits for the result BLPOP job:done:$job_id 0 and the server pushes the result into that queue for the client to consume.
If during the servicing of the job, e.g. once the server has called BLPOP and got the job details,the server dies, the client would be waiting forever for a response, but it would never arrive.
Is there some way to detect this condition within Redis, e.g. after the server accepts the job, crashes, the connection is closed (e.g. the tcp socket is closed by the OS). Can Redis signal that a client has disconnected and can we use that to detect that the job was accepted but will never be completed? Or perhaps there is a better mechanism for RPC within redis other than queues?
What you usually do is actually have your popping client do a BRPOPLPUSH from the original list to a special "I'm handling this list". Periodically, you can have another process scan the special list to detect "lost" jobs and return them to the queue (or whatever other policy you wish to implement).