laravelqueuebeanstalkd

Remove delay in Laravel Queue


I'm fan of Laravel-Lumen frameworks. These are so good and nice designed and we can start to implement the apps. But there is a small problem. May be this isn't a problem but in my opinion this is a small problem. Let me explain it.

For example I have a model and I'm using elastic search. When a model created (insert to db) then I'm dispatching a job for indexing this model.

public static function boot()
{
    parent::boot();

    static::created(function ($model) {
        $modelClass = get_class($model);
        lgi('>>> model created:' . $modelClass, $model);

        dispatch(new ElasticIndexerJob('creted', $model));
    });

    static::updated(function ($model) {
        $modelClass = get_class($model);
        lgi('>>> model updated:' . $modelClass, $model);

        dispatch(new ElasticIndexerJob('updated', $model));
    });

    static::deleted(function ($model) {
        $modelClass = get_class($model);
        lgi('>>> model deleted:' . $modelClass, $model);

        dispatch(new ElasticIndexerJob('deleted', $model));
    });
}

This outputs that log:

[2021-04-18 11:28:36] local.INFO: /app/Models/Traits/Indexable.php:25 [
    ">>> model updated:App\\Models\\City",
    {
        "id": 3,
        "country_id": 85,
        "state_id": 1,
        "zip_code": "87506",
        "name": "Nambe",
        "lat": 35.8890389,
        "lng": -106.0657318,
        "status": "passive",
        "created_at": "2021-04-13 09:38:09",
        "updated_at": "2021-04-18 11:28:36"
    }
]

The time is 11:28:36. After that I'm looking to queue log output.

[2021-04-18 11:28:38][27] Processing: App\Jobs\ElasticIndexerJob
[2021-04-18 11:28:38] local.INFO: /app/Jobs/ElasticIndexerJob.php:32 [
    "App\\Models\\City",
    "updated",
    {
        "id": 3,
        "country_id": 85,
        "state_id": 1,
        "zip_code": "87506",
        "name": "Nambe",
        "lat": 35.8890389,
        "lng": -106.0657318,
        "status": "passive",
        "created_at": "2021-04-13 09:38:09",
        "updated_at": "2021-04-18 11:28:36"
    }
]  
[2021-04-18 11:28:38][27] Processed:  App\Jobs\ElasticIndexerJob

Queue output time is 11:28:38. As you can see there is a 2 seconds delay (or difference). This problem is occuring all queues (sending mails, executing other jobs etc...). When sending mails this isn't a problem but sometimes I need realtime execution. I want to execute job immediatelly when I dispatch.

Normally if I use beanstalkd myself then I can receive messages (jobs) immediatelly and I can make something with that message. But in Laravel (or Lumen) there is a delay and this is annoying me.

Why Laravel has delay and how can I remove this delay?

I tested this in database queue and beanstalkd queue. Same behaviour occurs:

QUEUE_CONNECTION=beanstalkd
; QUEUE_CONNECTION=database

Note: lgi() function is an envelope of Log::info()


Solution

  • I found the solution. I look to Laravel's source code and I see that code in this file: Illuminate\Queue\Listener

    /**
     * The amount of seconds to wait before polling the queue.
     *
     * @var int
     */
    protected $sleep = 3;
    

    I investigated a little more deeply and I found that:

    /**
     * Create the command with the listener options.
     *
     * @param  string  $connection
     * @param  string  $queue
     * @param  \Illuminate\Queue\ListenerOptions  $options
     * @return array
     */
    protected function createCommand($connection, $queue, ListenerOptions $options)
    {
        return array_filter([
            $this->phpBinary(),
            $this->artisanBinary(),
            'queue:work',
            $connection,
            '--once',
            "--name={$options->name}",
            "--queue={$queue}",
            "--backoff={$options->backoff}",
            "--memory={$options->memory}",
            "--sleep={$options->sleep}",
            "--tries={$options->maxTries}",
        ], function ($value) {
            return ! is_null($value);
        });
    }
    

    As you can see there is a CLI param which is --sleep=[second]. We can set this value to zero (0). After that queue worker not waiting for process the next job.

    php artisan queue:work --sleep=0.1
    

    I think these parameters must be in documentation but Laravel developers didn't add this. Or may be I missed this I don't know.

    Edit: If you're using database as queue backend then this usage can load up your db. Becouse db driver is sending SELECT query to jobs table. I suggest that use beanstalkd driver with this usage.

    Edit 2: If you set --sleep to 0 (zero) then it may eat the cpu.

    Edit 3: I see this now. Probably I must go to a ophthalmologist :S https://laravel.com/docs/8.x/queues#worker-sleep-duration