I'm having a problem where the response I get from nginx is truncated, meaning that the source returned just ends part way through.
If I run the nginx container locally everything works, the problem does not happen until I move nginx into Azure. The significant environment difference that I can think of is the load balancer that Azure creates but I don't know much about how to troubleshoot that. In the nginx logs I see the following:
php access log
10.0.0.47 - - [29/Jan/2025:01:07:57 +0000] "GET /s/contacts HTTP/1.1" 200 65365 "https://mautic-nginx.thankfulmoss-a713c7bc.eastus2.azurecontainerapps.io/s/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.47 - - [29/Jan/2025:01:08:06 +0000] "GET /s/dashboard HTTP/1.1" 200 65381 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.47 - - [29/Jan/2025:01:20:07 +0000] "GET /s/keep-alive?_=1738112884973&mauticUserLastActive=721 HTTP/1.1" 200 5 "https://mautic-nginx.thankfulmoss-a713c7bc.eastus2.azurecontainerapps.io/s/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.47 - - [29/Jan/2025:01:32:07 +0000] "GET /s/keep-alive?_=1738112884974&mauticUserLastActive=1441 HTTP/1.1" 200 5 "https://mautic-nginx.thankfulmoss-a713c7bc.eastus2.azurecontainerapps.io/s/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.47 - - [29/Jan/2025:01:39:09 +0000] "GET /s/dashboard HTTP/1.1" 200 138978 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.47 - - [29/Jan/2025:01:42:16 +0000] "GET / HTTP/1.1" 302 306 "https://sandbox-4.reactblade.portal.azure.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.47 - - [29/Jan/2025:01:42:16 +0000] "GET /s/dashboard HTTP/1.1" 200 65397 "https://sandbox-4.reactblade.portal.azure.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.47 - - [29/Jan/2025:02:01:00 +0000] "GET / HTTP/1.1" 302 306 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.47 - - [29/Jan/2025:02:01:01 +0000] "GET /s/dashboard HTTP/1.1" 200 65389 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
static access log
10.0.0.11 - - [28/Jan/2025:19:24:09 +0000] "GET /media/libraries/ckeditor/ckeditor.js?v58771f16 HTTP/1.1" 200 1587514 "https://mautic-nginx.thankfulmoss-a713c7bc.eastus2.azurecontainerapps.io/s/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.11 - - [28/Jan/2025:19:24:10 +0000] "GET /app/bundles/CoreBundle/Assets/css/libraries/remixicon/fonts/remixicon.woff2?t=1718271040674 HTTP/1.1" 200 165440 "https://mautic-nginx.thankfulmoss-a713c7bc.eastus2.azurecontainerapps.io/media/css/libraries.css?v58771f16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.11 - - [28/Jan/2025:19:24:10 +0000] "GET /app/assets/images/favicon.ico?v58771f16 HTTP/1.1" 200 12862 "https://mautic-nginx.thankfulmoss-a713c7bc.eastus2.azurecontainerapps.io/s/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
10.0.0.47 - - [29/Jan/2025:02:01:07 +0000] "GET /favicon.ico HTTP/1.1" 200 12862 "https://mautic-nginx.thankfulmoss-a713c7bc.eastus2.azurecontainerapps.io/healthz" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
Error log
2025/01/29 02:52:17 [info] 14#14: *1282 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:22 [info] 14#14: *1283 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:27 [info] 11#11: *1284 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:27 [info] 11#11: *1285 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:32 [info] 14#14: *1286 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:37 [info] 11#11: *1288 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:37 [info] 14#14: *1287 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:42 [info] 14#14: *1289 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:47 [info] 11#11: *1291 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:47 [info] 14#14: *1290 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:52 [info] 14#14: *1292 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:57 [info] 11#11: *1294 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:52:57 [info] 11#11: *1293 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:53:02 [info] 14#14: *1295 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:53:07 [info] 14#14: *1296 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:53:07 [info] 14#14: *1297 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
2025/01/29 02:53:12 [info] 11#11: *1298 client closed connection while waiting for request, client: 10.0.0.33, server: 0.0.0.0:80
The client closed connection errors I am sure are coming from the load balancer health checks. Which I don't understand because when I hit the health check it returns 200 "healthy".
Where do I look from here? Which configuration files would likely have my issue in it? or is it more likely that the issue is with the load balancer?
What I found after a lot of digging around is that my nginx configuration was not taking into account the load balancer as a pool.
In my http block I needed to use upstream to identify the backend server addresses that are used for load balancing:
upstream mauticWeb {
least_conn;
server ${MAUTIC_WEB_URL}:9000;
}
and in my http --> server block wherever I needed to reach the backend php-fpm server I needed to use the upstream config.
server{
...
location ~ ^(.+\.php)(.*)$ {
...
fastcgi_keep_conn on;
fastcgi_pass mauticWeb;
...
}
...
}