Why is nginx not serving .well-known for certbot challenge?

there's a ton of similar questions, but I simply cannot figure out why my nginx config does not return anything for the acme challenge. Domain changed to example.com for the question.

My config looks like this:

server {    
    listen 443 ssl;
    server_name example.com;
    server_tokens off;
    ssl_certificate /opt/bitnami/letsencrypt/certificates/example.com.crt;
    ssl_certificate_key /opt/bitnami/letsencrypt/certificates/example.com.key;
    location / {        
        proxy_pass http://localhost:9000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }}
    
    server {    
        listen 80;
        server_name example.com;
        location ^~ /.well-known/acme-challenge/ {        
            alias /opt/bitnami/apps/letsencrypt/.well-known/acme-challenge/;
        }     
        return 301 https://$server_name$request_uri;
}

The actual renew script gives me this output:

$ sudo /opt/bitnami/letsencrypt/lego --path /opt/bitnami/letsencrypt --email="me@example.com" --http --http-timeout 30 --http.webroot /opt/bitnami/apps/letsencrypt --domains=example.com --user-agent bitnami-bncert/2.0.0 renew2024/09/21
15:06:04 [INFO] [example.com] acme: Trying renewal with 64 hours remaining2024/09/21 
15:06:04 [INFO] [example.com] acme: Obtaining bundled SAN certificate2024/09/21 
15:06:05 [INFO] [example.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4065531866462024/09/21 
15:06:05 [INFO] [example.com] acme: Could not find solver for: tls-alpn-012024/09/21 
15:06:05 [INFO] [example.com] acme: use http-01 solver2024/09/21 
15:06:05 [INFO] [example.com] acme: Trying to solve HTTP-012024/09/21 
15:06:13 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4065531866462024/09/21 
15:06:13 error: one or more domains had a problem:[example.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 13.53.80.234: Invalid response from https://example.com/.well-known/acme-challenge/7yobUqAmxObbYyWqgWq8znLTEbiRfSNsurCP8WxjVVA: 404

Where I'm wondering about the last line, particularly that it is a https. I'm thinking the server returns nothing for the http version of that URL, thus redirects to https, and then does not find anything.

I've added a dummy file to /opt/bitnami/apps/letsencrypt/.well-known/acme-challenge/test.html just to see if http://example.com/.well-known/acme-challenge/test.html would return anything, but same https 404 page.

Is the issue that nothing being written into the acme-challange directory, or a setup with the nginx http?

The problem is the return statement in the same block as the location statement. The return is evaluated first.

The solution is to wrap the return inside its own location block.

For example:

server {    
    listen 80;
    server_name example.com;
    location ^~ /.well-known/acme-challenge/ {        
        root /opt/bitnami/apps/letsencrypt/;
    }
    location {
        return 301 https://$server_name$request_uri;
    }
}