Serve pre-compressed cached files in Nginx and Passenger

I'm running a Rails App on Passenger with Nginx. Some of my App's files are mostly static so I cache them with page caching which writes a .html file and a .html.gz pre-compressed version in the public directory for Nginx to serve the next time it's requested. Nginx serves the .html file correctly if it is present and passes the request to Passenger if it's not. It doesn't seem to be serving the .html.gz file though.

I have compiled nginx with the needed modules --with-http_gunzip_module --with-http_gzip_static_module

Nginx config important bits:

http {
  gzip on;
  gzip_static on;
  gunzip on;
}

.html.gz files are served correctly when I'm just using Nginx to serve static sites. When I enable Passenger with passenger_enabled on; it doesn't serve the .html.gz file but just compresses the .html file on the fly and serves that.

Redacted server block:

server {
    listen          0.0.0.0:443 ssl http2;
    server_name     example.com;
    root /home/example/www.example.com/current/public;
    access_log  /home/example/log/access.log;
    
    ##
    # Nginx Bad Bot Blocker Includes
    # REPO: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
    ##
    include /etc/nginx/bots.d/ddos.conf;
    include /etc/nginx/bots.d/blockbots.conf;

    ssl_certificate      /etc/letsencrypt/live/www.example.com/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/live/www.example.com/privkey.pem;

    # Provide as much of Apache's SSL +StdEnvVars data as possible.
    fastcgi_param HTTPS                 on;
    fastcgi_param SSL_PROTOCOL          $ssl_protocol;
    fastcgi_param SSL_CIPHER            $ssl_cipher;
    fastcgi_param SSL_SESSION_ID        $ssl_session_id;
    fastcgi_param SSL_CLIENT_VERIFY     $ssl_client_verify;  

    passenger_enabled on;

    location ~* ^(.+)\.(png|jpe?g)$ {
        set $base $1;
        set $avif_uri $base$avif_suffix; # image.avif
        set $avif_with_orig_uri $base.$2.$avif_suffix; # image.png.avif
    
        expires 1y;
        add_header Cache-Control public;
        add_header ETag "";
        add_header Vary Accept;
        log_not_found off;
        try_files $avif_uri $avif_with_orig_uri $uri =404;    
    }
    location ~ ^/assets/ {
        expires 1y;
        add_header Cache-Control public;
        add_header ETag "";
        break;
    }
}

Removing passenger_enabled on; from the server block and adding:

location / {
    try_files $uri.html $uri @passenger;
}
location @passenger {
    passenger_enabled on;
}

Thanks to Camden Narzt and Frank Groeneveld for pointing me in the right direction