nginxamazon-eksnlb

Why am I getting "upstream server temporarily disabled while reading response header from upstream"?

I have deployed NGINX Plus Ingress controller on AWS EKS, in front of the ingress I have deployed a network load balancer (NLB). I have set up a user pool in AWS Cognito and set up JWT validation in NGINX according to the instructions here https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-jwt-authentication/#getting-jwks-from-subrequest.

auth_jwt             "closed";
auth_jwt_type        signed;
auth_jwt_key_cache   1h;
auth_jwt_key_request /jwks_uri;

...

location = /jwks_uri {
internal;
proxy_pass https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json;
}

I have tried all kinds of proxy settings - everything from increasing time outs, setting proxy_method GET, stripping headers but nothing seems to help. I also cut the sub request part out of the equation by just adding

location /jwks_uri {
proxy_pass https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json;          
}

Requests there succeed, so I don't think it's a firewall issue.

When I fire off the request with a valid JWT issued by Cognito in the Authorization header, if I wait more than a minute I eventually get a 502 response. In the NGINX logs I can see

2022/11/24 08:47:55 [warn] 56#56: *19 upstream server temporarily disabled while reading response header from upstream, client: 10.10.20.57, server: <server>, request: "POST /graphql HTTP/1.1", subrequest: "/jwks_uri", upstream: <jwks-uri>", host: <host

This seems to happen when NGINX attempts the sub request to fetch jwks keys from AWS from https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json

Solution

  • The original request was a POST and I needed to add the following proxy settings in order for the subrequest to sufficiently convert to a GET

    location = /jwks_uri {
        internal;
        proxy_method GET;
        proxy_set_header Host {idp}.{region}.amazonaws.com;
        proxy_set_header Content-Length "";
        proxy_pass https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json;
      }

    Posting in case someone else runs into the same issue.