[SOLVED] Laravel queued job fails with different exceptions after 5-20 minutes of execution

The job is loading a huge 100M CSV into mysql table via LOAD DATA INFILE.

After different periods of working the job fails

App\Jobs\UpdateProducts ................... 17m 43s FAIL

but remains in jobs table and doesn't move to failed_jobs.

Errors in worker.log every time difer:

[previous exception] [object] (PDOException(code: 2002): SQLSTATE[HY000] [2002] No such file or directory at /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:65)


[2024-07-12 22:02:12] local.ERROR: SQLSTATE[HY000] [2002] No such file or directory (Connection: mysql, SQL: select * from `cache` where `key` = illuminate:queue:restart limit 1) {"exception":"[object] (Illuminate\\Database\\QueryException(code: 2002): SQLSTATE[HY000] [2002] No such file or directory (Connection: mysql, SQL: select * from `cache` where `key` = illuminate:queue:restart limit 1) at /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php:813)
[stacktrace]
#0 /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php(967): Illuminate\\Database\\Connection->runQueryCallback()
#1 /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php(946): Illuminate\\Database\\Connection->tryAgainIfCausedByLostConnection()
#2 /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php(769): Illuminate\\Database\\Connection->handleQueryException()

SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: select * from `cache` where `key` = illuminate:queue:restart limit 1)

[2024-07-12 21:56:30] local.ERROR: SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: select * from `cache` where `key` = illuminate:queue:restart limit 1) {"exception":"[object] (Illuminate\\Database\\QueryException(code: 2002): SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: select * from `cache` where `key` = illuminate:queue:restart limit 1) at /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php:813)
[stacktrace]
#0 /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php(967): Illuminate\\Database\\Connection->runQueryCallback()
#1 /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php(946): Illuminate\\Database\\Connection->tryAgainIfCausedByLostConnection()


Next Illuminate\\Database\\QueryException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (Connection: mysql, SQL: 
                    LOAD DATA INFILE '/var/lib/mysql-files/products.csv' INTO TABLE products
                    FIELDS TERMINATED BY ','
                    OPTIONALLY ENCLOSED BY '\"'
                    LINES  TERMINATED BY '
'
                    (name, description, price, deleted, top)
                ) in /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php:813
Stack trace:
#0 /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php(767): Illuminate\\Database\\Connection->runQueryCallback()
#1 /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php(560): Illuminate\\Database\\Connection->run()
#2 /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/DatabaseManager.php(456): Illuminate\\Database\\Connection->statement()


#43 {main}, 2024-07-12 21:56:27)) {"exception":"[object] (Illuminate\\Database\\QueryException(code: 2002): SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: insert into `failed_jobs` (`uuid`, `connection`, `queue`, `payload`, `exception`, `failed_at`) values (c3e93029-82ef-4af9-93f5-ae3ab1ddef68, database, default, {\"uuid\":\"c3e93029-82ef-4af9-93f5-ae3ab1ddef68\",\"displayName\":\"App\\\\Jobs\\\\UpdateProducts\",\"job\":\"Illuminate\\\\Queue\\\\CallQueuedHandler@call\",\"maxTries\":1,\"maxExceptions\":null,\"failOnTimeout\":true,\"backoff\":null,\"timeout\":36000,\"retryUntil\":null,\"data\":{\"commandName\":\"App\\\\Jobs\\\\UpdateProducts\",\"command\":\"O:23:\\\"App\\\\Jobs\\\\UpdateProducts\\\":0:{}\"}}, PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php:571
Stack trace:
#0 /home/www/web/tinytest/vendor/laravel/framework/src/Illuminate/Database/Connection.php(571): PDOStatement->execute()

supervisor config:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/web/tinytest/artisan queue:work --sleep=30
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=1
redirect_stderr=true
stdout_logfile=/var/www/web/tinytest/storage/logs/worker.log
stopwaitsecs=3600

my.cnf

[mysqld]
innodb_file_per_table=ON
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=3G
max_allowed_packet = 256M

mysql-client also seems to reconnect after fail so server goes away indeed it looks like.

There has been no such issue on the mariadb (same code and data).