nginxgoogle-cloud-platformgoogle-cloud-rungoogle-vpc

Nginx in Cloud Run with internal traffic works but gives connect errors


I'm running an Nginx in a Cloud Run instance as a Reverse-Proxy to a backend Cloud Run app (it will do more stuff in the future, but let's keep it simple for this example).

The Nginx Cloud Run requires authentication (via IAM), but the backend app doesn't. The Nginx is connected to the same VPC and has the setting (vpc_access_egress = all-traffic), and the backend app is set to Allow internal traffic only only.

events {}
http {
    resolver 169.254.169.254;

    server {
        listen 8080;
        server_name mirror_proxy;

        location / {
            proxy_pass https://my-backend.a.run.app:443;
            proxy_cache off;
            proxy_redirect off;
        }
    }
}

The setup works, and I send authenticated requests to the Nginx and get the responses from the backend. However I also get a lot of error messages from the Nginx per request.

2022/12/22 13:57:51 POST 200 1.76 KiB 1.151s curl 7.68.0 https://nginx.a.run.app/main
2022/12/22 13:57:50 [error] 4#4: *21 connect() to [1234:5678:4802:34::35]:443 failed
  (113: No route to host) while connecting to upstream, client: 169.254.1.1,
  server: mirror_proxy, request: "POST /main HTTP/1.1",
  upstream: "https://[1234:5678:4802:34::35]:443/main", host: "nginx.a.run.app"
2022/12/22 13:57:50 [error] 4#4: *21 connect() to [1234:5678:4802:36::35]:443 failed
  (113: No route to host) while connecting to upstream, client: 169.254.1.1,
  server: mirror_proxy, request: "POST /main HTTP/1.1",
  upstream: "https://[1234:5678:4802:36::35]:443/main", host: "nginx.a.run.app"
2022/12/22 13:57:50 [error] 4#4: *21 connect() to [1234:5678:4802:32::35]:443 failed
  (113: No route to host) while connecting to upstream, client: 169.254.1.1,
  server: mirror_proxy, request: "POST /main HTTP/1.1",
  upstream: "https://[1234:5678:4802:32::35]:443/main", host: "nginx.a.run.app"

Why are there errors, when the request succeeds? Doesn't the VPC router don't know the exact IP address of the Cloud Run yet, and Nginx has to try them out? Any idea?


Solution

  • GCP only uses IPv4 inside the VPC network. Since I forced the Nginx to use the VPC network (vpc_access_egress = all-traffic), Nginx will fail when it tries to resolve an IPv6, and fall back to IPv4.

    With the following setting you can force Nginx to immediately resolve the IPv4.

    http {
        resolver 169.254.169.254 ipv6=off;
    
        ...
    }
    
    ``