Is there any rule of thumb or any logical relation between maxProcesses, number of supervisors and the total number of queues in laravel horizon?
What if I have 15 supervisors and 40 queues (each supervisor has multiple queues based on their category)? What is the maximum number of maxProcesses I can assign to each supervisor (suppose balancing auto)?
I want to know that if there's a rule of thumb for a better performance on horizon by tuning these numbers, for example if the number of supervisor-x should not exceed the total number of queues and if the maxProcesses should not exceed a certain number based on the OS spec running the processes.
Is there any logical relation between these numbers? Is there a good document about this issue? I have seen this document on supervisor and also the Laravel Horizon docs, but have not found the answer to my questions.
I need to explain things in detail in order to understand the relation between all these things.
Supervisor exists out of some simple settings. The most important once are these:
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/forge/app.com/artisan queue:work
autostart=true
autorestart=true
numprocs=8
The most important setting here is numprocs=8, from supervisor the manual it says:
Supervisor will start as many instances of this program as named by numprocs. Note that if numprocs > 1, the process_name expression must include %(process_num)s (or any other valid Python string expression that includes process_num) within it.
This configuration of supervisor running a program called artisan queue:work will create 8 instances (processes, workers, the same thing) of artisan queue:work. This means that 8 jobs can be processed simultaneously, nothing more, nothing less.
Horizon doesn't define the numprocs, the only important setting you'll have to know is the stopwaitsecs=3600. This should always be far greater than the maximum time a job runs in your entire application. Here the absolute maximum amount would be 60 minutes.
Now Horizon comes with a balancing strategy where you can define the min and max number of processes (workers) and it's strategy using
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
What Horizon offers to do here is scale up or down the amount of processes (workers) according to the amount of workload present in the queue(s).
If you define a supervisor configuration like the following:
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default', 'events', 'xls', 'whatever'],
'balance' => 'auto',
'minProcesses' => 10,
'maxProcesses' => 40,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'tries' => 3,
],
],
],
Then all 4 queues, default, events, xls and whatever run all under the same conditions, will have a total of 40 workers available and a minimum of 10. So not each queue has 40 workers available, but all combined have 40 workers (processes) available.
The key point here of getting a good scale for each queue to work optimally, is to divide them into different categories, e.g.
short-load -> each job takes about 1 to 5 seconds.medium-load -> each job takes about 5 to 30 seconds.long-load -> each job takes up to 5 minutes.extreme-load -> each job takes longer than 5 minutes, up to an hour.If you only end up with two scenarios, like short-load and long-load, then you will have two configurations for horizon in such a way which would define how fast supervisor will respond to spawn new workers and how many times it will try to repeat a job if it has failed (where you seriously don't want to try a job that will fail each time after 59 minutes 3 times).
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default', 'events'],
'balance' => 'auto',
'minProcesses' => 10,
'maxProcesses' => 40,
'balanceMaxShift' => 10,
'balanceCooldown' => 1,
'tries' => 3,
],
'supervisor-long-run' => [
'connection' => 'redis',
'queue' => ['xls', 'whatever'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'tries' => 1,
],
],
],
In one of your last comments you asked
I want to understand all those calculations you make, what's the formula for it
The formula is, 1 supervisor instance can have many queues, and all of these queues have a maximum amount of workers available. The queues are not that important, but the amount of jobs (and the kind of jobs) placed in these queues in a certain amount of time is.
Example:
4 queues producing 120 jobs each minute, need x amount of workers to be processed. If you scale up (or down) the amount of workers (processes), the amount of time it takes to process all these jobs until the queues are empty relates to the amount of workers you make available.
If 1 job takes 10 seconds to complete (as an example average) and an average of 120 jobs are put on a queue each minute. If you would like to process (clear the queue) all jobs within one minute, you need 120 jobs * 10 seconds for each job / 60 seconds in a minute = the amount of workers (processes) needed to complete all those jobs within 1 minute.
So yes, you can scale up the amount of workers to 64, 512 or 24890. It comes all back to the question how much load can your hardware handle.
Hope it made sense.
I'll clean up the text tomorrow using only workers, processes or instances .. it's such a mess ;)