I have a test server that is running two versions of the exact same Laravel application each with their own instance of Laravel Horizon running (A staging site for new features and a demo site that mirrors production). These applications are running in docker containers and they share the same Redis DB. IE: There is one Redis docker container, but two separate PHP containers running Laravel Horizon. They have been running on the same server for several months but this issue just popped in the past week.
When this issue first popped up I realized didn't have the HORIZON_PREFIX env set for each site. This has been corrected with stg: and demo: as the unique env vars.
I did a FLUSHDB on the redis DB and restarted all docker containers. Restarting the docker container runs this script to reset the application cache as well.
echo 'Clearing previous cache...'
(cd /var/www/html && php artisan cache:clear && php artisan config:clear && php artisan route:clear && php artisan view:clear)
echo "Caching configuration..."
(cd /var/www/html && php artisan config:cache && php artisan route:cache && php artisan view:cache)
echo "Starting Laravel application..."
But still when I submit a job it is sent to both the stg & demo queues regardless that I have unique HORIZON_PREFIX values.
This is what's contained in my redis DB; you can see there's separation from demo & stg, but the issue remains. Jobs get submitted to both systems. Is there some other step I need to prevent this collision?
You can see two failed jobs below. Those were jobs submitted to the demo site. They got sent to both Horizon instances; failed on the staging site and still stuck in the pending jobs list on demo. If I turn off the staging site and submit the job again, it completes in 5 seconds successfully.
127.0.0.1:6379> KEYS *
1) "demo:monitor:time-to-clear"
2) "demo:measured_jobs"
3) "stg:supervisor:7e9026e6c9b7-6nE4:supervisor-2"
4) "demo:db636ed0-c5ac-4233-9867-aed91e9397af"
5) "demo:masters"
6) "stg:last_snapshot_at"
7) "stg:monitor:time-to-clear"
8) "demo:master:3a70f005424b-Bteh"
9) "stg:supervisor:7e9026e6c9b7-6nE4:supervisor-0"
10) "stg:failed:App\\Models\\CustomList:79"
11) "demo:queue:default"
12) "stg:masters"
13) "demo:last_snapshot_at"
14) "demo:9fcc3255-cb91-4ad6-ada1-24528c806b63"
15) "demo:pending_jobs"
16) "stg:failed_jobs"
17) "stg:failed:App\\Models\\CustomList:48"
18) "stg:recent_failed_jobs"
19) "stg:9fcc3255-cb91-4ad6-ada1-24528c806b63"
20) "demo:supervisor:3a70f005424b-Bteh:supervisor-0"
21) "stg:supervisor:7e9026e6c9b7-6nE4:supervisor-1"
22) "demo:measured_queues"
23) "stg:master:7e9026e6c9b7-6nE4"
24) "stg:db636ed0-c5ac-4233-9867-aed91e9397af"
25) "stg:6d53039b-dcd9-41f6-af5e-3c89203dd6e7"
26) "demo:completed_jobs"
27) "demo:supervisors"
28) "demo:supervisor:3a70f005424b-Bteh:supervisor-2"
29) "stg:supervisors"
30) "demo:d96f69ba-60fd-4d2f-a8c0-1a8091cb98bf"
31) "demo:job:App\\Jobs\\SyncData"
32) "demo:supervisor:3a70f005424b-Bteh:supervisor-1"
33) "demo:0705c7ec-d81e-4e26-9f40-d5c045e33c5b"
34) "stg:b159e41e-fb2e-40f9-9c3e-c404deb1bacd"
35) "demo:6d53039b-dcd9-41f6-af5e-3c89203dd6e7"
36) "demo:recent_jobs"
37) "demo:b159e41e-fb2e-40f9-9c3e-c404deb1bacd"
127.0.0.1:6379>
VERSIONS
Laravel Horizon v5.16.1
Laravel v9.52.9
For anyone that has this issue, I found the CACHE_PREFIX setting in the config/cache.php file. When I set unique values here as well, my issue went away.