How to launch a Symfony Command asynchronously during a request using the Symfony Process component?

I'm trying to execute a Symfony Command using the Symfony Process component so it executes asynchronously when getting an API request.

When I do so I get the error message that Code: 127(Command not found), but when I run it manually from my console it works like a charm.

This is the call:

public function asyncTriggerExportWitnesses(): bool
{
    $process = new Process(['php /var/www/bin/console app:excel:witness']);
    $process->setTimeout(600);
    $process->setOptions(['create_new_console' => true]);
    $process->start();
    $this->logInfo('pid witness export: ' . $process->getPid());
    if (!$process->isSuccessful()) {
        $this->logError('async witness export failed: ' . $process->getErrorOutput());
        throw new ProcessFailedException($process);
    }

    return true;
}

And this is the error I get:

The command \"'php /var/www/bin/console app:excel:witness'\" failed.
Exit Code: 127(Command not found)
Working directory: /var/www/public
Output:================
Error Output:================
sh: exec: line 1: php /var/www/bin/console app:excel:witness: not found

What is wrong with my usage of the Process component?

Calling it like this doesn't work either:

$process = new Process(['/usr/local/bin/php', '/var/www/bin/console', 'app:excel:witness']);

this results in following error:

The command \"'/usr/local/bin/php' '/var/www/bin/console' 'app:excel:witness'\" failed.
Exit Code: ()
Working directory: /var/www/public
Output:
================
Error Output:
================

Solution

First, note that the Process component is not meant to run asynchronously after the parent process dies. So triggering async jobs to run during an API request is a not a good use case.

These two comments in the docs about running things asynchronously are very pertinent:

If a Response is sent before a child process had a chance to complete, the server process will be killed (depending on your OS). It means that your task will be stopped right away. Running an asynchronous process is not the same as running a process that survives its parent process.

If you want your process to survive the request/response cycle, you can take advantage of the kernel.terminate event, and run your command synchronously inside this event. Be aware that kernel.terminate is called only if you use PHP-FPM.

Beware also that if you do that, the said PHP-FPM process will not be available to serve any new request until the subprocess is finished. This means you can quickly block your FPM pool if you’re not careful enough. That is why it’s generally way better not to do any fancy things even after the request is sent, but to use a job queue instead.

If you want to run jobs asynchronously, just store the job "somewhere" (e.d a database, redis, a textfile, etc), and have a decoupled consumer go through the "pending jobs" and execute whatever you need without triggering the job within an API request.

This above is very easy to implement, but you could also just use Symfony Messenger that will do it for you. Dispatch messages on your API request, consume messages with your job queue consumer.

All this being said, your use of process is also failing because you are trying mixing sync and async methods.

Your second attempt at calling the command is at least successful in finding the executable, but since you call isSuccessful() before the job is done.

If you use start() (instead of run()), you cannot simply call isSuccessful() directly, because the job is not finished yet.

Here is how you would execute an async job (although again, this would very rarely be useful during an API request):

class ProcessCommand extends Command
{

    protected static $defaultName = 'process_bg';

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $phpBinaryFinder = new PhpExecutableFinder();

        $pr = new Process([$phpBinaryFinder->find(), 'bin/console', 'bg']);
        $pr->setWorkingDirectory(__DIR__ . '/../..');

        $pr->start();

        while ($pr->isRunning()) {
            $output->write('.');
        }
        $output->writeln('');

        if ( ! $pr->isSuccessful()) {
            $output->writeln('Error!!!');

            return self::FAILURE;
        }

        $output->writeln('Job finished');

        return self::SUCCESS;
    }

}