How to prepend Sidekiq job to beginning of a queue?
I know that most obvious thing you might want to say right now is "just add a separate high-prio queue, with a separate worker!", but I'm constrained on resources, so an extra worker is out of the question.
Now, I know that technically Sidekiq is prepending jobs to the queue in Redis, and pops them from the end of the list to get next one for processing (which is somewhat counterintuitive, but ok). Occasionally, to preserve unfinished jobs when terminating, it actually does what I need: it re-queues jobs to the end of the list (== they will be pulled next). So, in the end, does anybody know how to prepend (well, technically, it's "append") a job to the queue so it's performed by the next fetch? Without monkey-patching the Sidekiq source too much, of course. Or, maybe, someone knows a gem that already does that?..
I'm constrained on resources, so an extra worker is out of the question.
You don't need an extra worker, you just have to configure a second queue. Even a single Sidekiq process with a single thread can process multiple queues.
Sidekiq queues are processed in the order given (or by weight, see Advanced Options – Queues).
Given this simple config:
:queues:
- critical
- default
An available Sidekiq worker would try to fetch jobs from the critical queue first and – if it's empty – from the default queue second.
Say you have 3 jobs in the default queue and none in critical
critical: []
default: [job_1, job_2, job_3]
... the next available worker would process job_1 first, then job_2, and finally job_3.
If you push another job to critical, i.e.:
critical: [job_4]
default: [job_1, job_2, job_3]
... the next available worker would process job_4 first and then job_1, job_2, and job_3.