Please see update at the bottom regarding the Nginx version 1.23.0 being the cause
I've been using NGINX to generate image thumbnails for a number of years, however having just switched from Ubuntu 16 to Ubuntu 22 and Nginx 1.16 to nginx 1.23 it no longer works with paths that include spaces.
It is very likely that this is a configuration difference rather than something to do with the different versions, however as far as I can tell the NGINX configs are identical so possibly it is something to do with different Ubuntu/Nginx versions.
The error given when accessing a URL with a space in the path is simply "400 Bad Request".
There are no references to the request in either the access or error logs after 400 Bad Request is returned.
The nginx site config looks like this and was originally based off this guide
server {
server_name localhost;
listen 8888;
access_log /home/mysite/logs/nginx-thumbnails-localhost-access.log;
error_log /home/mysite/logs/nginx-thumbnails-localhost-error.log error;
location ~ "^/width/(?<width>\d+)/(?<image>.+)$" {
alias /home/mysite/$image;
image_filter resize $width -;
image_filter_jpeg_quality 95;
image_filter_buffer 8M;
}
location ~ "^/height/(?<height>\d+)/(?<image>.+)$" {
alias /home/mysite/$image;
image_filter resize - $height;
image_filter_jpeg_quality 95;
image_filter_buffer 8M;
}
location ~ "^/resize/(?<width>\d+)/(?<height>\d+)/(?<image>.*)$" {
alias /home/mysite/$image;
image_filter resize $width $height;
image_filter_jpeg_quality 95;
image_filter_buffer 8M;
}
location ~ "^/crop/(?<width>\d+)/(?<height>\d+)/(?<image>.*)$" {
alias /home/mysite/$image;
image_filter crop $width $height;
image_filter_jpeg_quality 95;
image_filter_buffer 8M;
}
}
proxy_cache_path /tmp/nginx-thumbnails-cache/ levels=1:2 keys_zone=thumbnails:10m inactive=24h max_size=1000m;
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name thumbnails.mysite.net;
ssl_certificate /etc/letsencrypt/live/thumbnails.mysite.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/thumbnails.mysite.net/privkey.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
access_log /home/mysite/logs/nginx-thumbnails-access.log;
error_log /home/mysite/logs/nginx-thumbnails-error.log error;
location ~ "^/width/(?<width>\d+)/(?<image>.+)$" {
# Proxy to internal image resizing server.
proxy_pass http://localhost:8888/width/$width/$image;
proxy_cache thumbnails;
proxy_cache_valid 200 24h;
}
location ~ "^/height/(?<height>\d+)/(?<image>.+)$" {
# Proxy to internal image resizing server.
proxy_pass http://localhost:8888/height/$height/$image;
proxy_cache thumbnails;
proxy_cache_valid 200 24h;
}
location ~ "^/resize/(?<width>\d+)/(?<height>\d+)/(?<image>.+)$" {
# Proxy to internal image resizing server.
proxy_pass http://localhost:8888/resize/$width/$height/$image;
proxy_cache thumbnails;
proxy_cache_valid 200 24h;
}
location ~ "^/crop/(?<width>\d+)/(?<height>\d+)/(?<image>.+)$" {
# Proxy to internal image resizing server.
proxy_pass http://localhost:8888/crop/$width/$height/$image;
proxy_cache thumbnails;
proxy_cache_valid 200 24h;
}
location /media {
# Nginx needs you to manually define DNS resolution when using
# variables in proxy_pass. Creating this dummy location avoids that.
# The error is: "no resolver defined to resolve localhost".
proxy_pass http://localhost:8888/;
}
}
I don't know if it's related but nginx-thumbnails-error.log also regularly has the following line, although testing the response in browser seems to work:
2022/07/19 12:02:28 [error] 1058111#1058111: *397008 connect() failed (111: Connection refused) while connecting to upstream, client: ????, server: thumbnails.mysite.net, request: "GET /resize/100/100/path/to/my/file.png HTTP/2.0", upstream: "http://[::1]:8888/resize/100/100/path/to/my/file.png", host: "thumbnails.mysite.net"
This error does not appear when accessing a file with a space in it.
There are no references to the request for a file with a space in the path in nginx-thumbnails-access.log or nginx-thumbnails-error.log.
But there is an entry in the access log for localhost nginx-thumbnails-localhost-access.log
./nginx-thumbnails-localhost-access.log:127.0.0.1 - - [29/Jul/2022:10:37:13 +0000] "GET /resize/200/200/test dir/KPjjCTl0lnpJcUdQIWaPflzAEzgN25gRMfAH5qiI.png HTTP/1.0" 400 150 "-" "-"
When a path has no spaces there is an entry in both nginx-thumbnails-localhost-access.log and nginx-thumbnails-access.log
./nginx-thumbnails-localhost-access.log:127.0.0.1 - - [29/Jul/2022:10:43:48 +0000] "GET /resize/200/202/testdir/KPjjCTl0lnpJcUdQIWaPflzAEzgN25gRMfAH5qiI.png HTTP/1.0" 200 11654 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
./nginx-thumbnails-access.log:185.236.155.232 - - [29/Jul/2022:10:43:48 +0000] "GET /resize/200/202/testdir/KPjjCTl0lnpJcUdQIWaPflzAEzgN25gRMfAH5qiI.png HTTP/2.0" 200 11654 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
I've no idea if it's relevant, but access log entries for images with a space in the name do not include the browser user agent.
./nginx-thumbnails-localhost-access.log:127.0.0.1 - - [29/Jul/2022:10:52:16 +0000] "GET /resize/200/202/testdir/thumb image.png HTTP/1.0" 400 150 "-" "-" ./nginx-thumbnails-localhost-access.log:127.0.0.1 - - [29/Jul/2022:10:52:33 +0000] "GET /resize/200/202/testdir/thumbimage.png HTTP/1.0" 200 11654 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
As requested here is a result from curl -i
HTTP/2 400
server: nginx
date: Thu, 28 Jul 2022 15:44:13 GMT
content-type: text/html
content-length: 150
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
I can now confirm that this is specific to Nginx 1.23.0 which is not yet stable.
I created a new Digital Ocean Droplet installed Nginx and setup the thumbnail server and it worked perfectly. The default nginx installation for Ubuntu 22 was 1.18.0.
I then upgraded to 1.23.0 as I had done on my live server via:
apt-add-repository ppa:ondrej/nginx-mainline -y
apt install nginx
The thumbnail server then stopped working with the same issue as my original issue.
I am now investigating downgrading nginx.
Downgrading Nginx to 1.18.0 worked, using these steps:
apt-add-repository --remove ppa:ondrej/nginx-mainline
apt autoremove nginx
apt update
apt install nginx
For some reason on one server I also had to run apt autoremove nginx-core but not on the other.
However i'm a little concerned that 1.18.0 is marked as no longer receiving security support. But I could not find an easy way to install 1.22.0 and test, only 1.18.0: https://endoflife.date/nginx
I figured out that this error was specific to Nginx version 1.23.0 and downgrading to 1.18.0 resolved the issue.
Downgrading Nginx to 1.18.0 worked, using these steps:
apt-add-repository --remove ppa:ondrej/nginx-mainline
apt autoremove nginx
apt update apt install nginx
For some reason on one server I also had to run apt autoremove nginx-core but not on the other.
I am still investigating whether 1.22.0 is okay and how to report this error to Nginx directly.
So thanks to this answer on the Nginx Bug Tracker: https://trac.nginx.org/nginx/ticket/1930
The solution was actually very simply to remove the URI components
From this:
location ~ "^/resize/(?<width>\d+)/(?<height>\d+)/(?<image>.+)$" {
# Proxy to internal image resizing server.
proxy_pass localhost:8888/resize/$width/$height/$image;
proxy_cache thumbnails;
proxy_cache_valid 200 24h;
}
To this:
location ~ "^/resize/(?<width>\d+)/(?<height>\d+)/(?<image>.+)$" {
# Proxy to internal image resizing server.
proxy_pass localhost:8888;
proxy_cache thumbnails;
proxy_cache_valid 200 24h;
}
And then it works.
This is the full explanation from @Maxim Dounin at the bug tracker:
That's because spaces are not allowed in request URIs, and since nginx 1.21.1 these are rejected. Quoting CHANGES:
*) Change: now nginx always returns an error if spaces or control characters are used in the request line.
See ticket #196 for details.
Your configuration results in incorrect URIs being used during proxying, as it uses named captures from $uri, which is unescaped, in proxy_pass, which expects all variables to be properly escaped if used with variables.
As far as I can see, in your configuration the most simple fix would be to change all proxy_pass directives to don't use any URI components, that is:
location ~ "/width/(?<width>\d+)/(?<image>.+)$" {
proxy_pass http://localhost:8888;
proxy_cache thumbnails;
proxy_cache_valid 200 24h;
}
This way, nginx will pass URIs from the client request unmodified (and properly escaped), and these match URIs you've been trying to reconstruct with variables.