pythonsupervisordpython-rq

Trying to get supervisor to create a worker for python-rq


I am trying to get supervisor to spawn a worker following this pattern using python-RQ, much like what is mentioned in this stackoverflow question. I can start workers manually from the terminal as follows:

$ venv/bin/rq worker
14:35:27 Worker rq:worker:fd403822518a4f21802fa0dc417e526a: started, version 1.2.2
14:35:27 *** Listening on default...

It works great. I can confirm the worker exists in another terminal:

$ venv/bin/rq info
0 queues, 0 jobs total

fd403822518a4f21802fa0dc417e526a (b'' 41735): idle default
1 workers, 0 queues

Now to start a worker using supervisor.... Here is my supervisord.conf file, located in the same directory.

[supervisord]

;[program:worker]
command=venv/bin/rq worker
process_name=%(program_name)s-%(process_num)s
numprocs=1
directory=.
stopsignal=TERM
autostart=false
autorestart=false

I can start supervisor as follows:

$ venv/bin/supervisord -n
2020-03-05 14:36:45,079 INFO supervisord started with pid 41757

However, checking for a new worker, I see it's not there.

$ venv/bin/rq info
0 queues, 0 jobs total

0 workers, 0 queues

I have tried a multitude of other ways to get this worker to start, such as...

... within the virtual environment:

$ source venv/bin/activate
(venv) $ rq worker
*** Listening on default...

... using a shell file

#!/bin/bash
source /venv/bin/activate
rq worker low

$ ./start.sh
*** Listening on default...

... using a python script

$ venv/bin/python3 worker.py
*** Listening on default...

When started manually they all work fine. Changing the command= in supervisord.conf doesn't seem to make a difference. There is no worker to be found. What am I missing? Why won't supervisor start a worker? I am running this in Mac OS and my file structure is as follows:

.
|--__pycache__
|--supervisord.conf
|--supervisord.log
|--supervisord.pid
|--main.py
|--task.py
|--venv
   |--bin
      |--rq
      |--supervisord
      |--...etc
   |--include
   |--lib
   |--pyenv.cfg

Thanks in advance.


Solution

  • I had two problems with supervisord.conf, which was preventing the worker from starting. The corrected config file is as follows:

    [supervisord]
    
    [program:worker]
    command=venv/bin/rqworker
    process_name=%(program_name)s-%(process_num)s
    numprocs=1
    directory=.
    stopsignal=TERM
    autostart=true
    autorestart=false
    

    First, the line [program:worker] was in fact commented out. I must have taken this line from the commented out sample file and not realized. However removing the comment still didn't start the worker.... I also had to set autostart=true, as starting supervisor does not automatically start a command.