I made a Symfony console command which uses pngquant
to process and compress a long list of images. The images are read from a CSV file.
The batch is working fine until the end in local environment but in the stage environment it works for about 5 minutes and then it starts returning empty result from the shell_exec
command. I even made a retry system but it's always returning empty result:
// escapeshellarg() makes this safe to use with any path
// errors are redirected to standard output
$command = sprintf(
'%s --quality %d-%d --output %s --force %s 2>&1',
$this->pngquantBinary,
$minQuality,
$maxQuality,
escapeshellarg($tempPath),
$path
);
// tries a few times
$data = null;
$attempt = 0;
do {
// command result
$data = shell_exec($command);
// error
if (null !== $data) {
$this->logger->warning('An error occurred while compressing the image with pngquant', [
'command' => $command,
'output' => $data,
'cpu' => sys_getloadavg(),
'attempt' => $attempt + 1,
'sleep' => self::SLEEP_BETWEEN_ATTEMPTS,
]);
sleep(self::SLEEP_BETWEEN_ATTEMPTS);
}
++$attempt;
} while ($attempt < self::MAX_NUMBER_OF_ATTEMPTS && null !== $data);
// verifies that the command has finished successfully
if (null !== $data) {
throw new \Exception(sprintf('There was an error compressing the file with command "%s": %s.', $command, $data));
}
The problem is that the same command executed in another shell in the same environment works fine! I mean, when I log the error, if I put the exactly same command in another instance on the same server, works fine.
Even from the Symfony logs I can't read any error, where should I look for a more detailed error?
What can be causing this? Memory and processor are fine during execution!
After many attempts I read this question:
Symfony2 Process component - unable to create pipe and launch a new process
The solution was to add a call to gc_collect_cycles
after flush during the loop!
if ($flush || 0 === ($index % self::BATCH_SIZE)) {
$this->em->flush();
$this->em->clear();
// clears the temp directory after flushing
foreach ($this->tempImages as $tempImage) {
unlink($tempImage);
}
$this->tempImages = [];
// forces collection of any existing garbage cycles
gc_collect_cycles();
}
IMPORTANT: also keep an eye on the number of disk inodes.
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 125193 392 124801 1% /dev
tmpfs 127004 964 126040 1% /run
/dev/vda2 5886720 5831604 55116 100% /