amazon-web-servicesnginxamazon-ecs

NGINX inside ECS suddenly pointing to other hostname


i have deployment using container at ECS that contain static web and nginx for reverse proxy. I'm using nginx:alpine as base image

Use Nginx to serve a static website and as a reverse proxy for another service inside ECS, which is behind a VPC.

I have an AWS load balancer in front of ECS.

/etc/nginx/nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    resolver 8.8.8.8 valid=60s;
    resolver_timeout 5s;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    upstream backend {
        server api.com:443;
        keepalive 32;
    }
#    proxy_read_timeout 10s;
#    proxy_connect_timeout 10s;
#    proxy_send_timeout 10s;
#    send_timeout 10s;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/default.conf

server {
    listen 3000;
    server_name web.com;

    location / {
        root /usr/share/nginx/html;
        index index.html;
        try_files $uri /index.html;
    }

    location /intools/ {
        client_max_body_size 50M;
        proxy_pass https://api.com;
        proxy_set_header Host api.com;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization $http_authorization;

        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_connect_timeout 10s;
        proxy_send_timeout 10s;
        proxy_read_timeout 10s;
        send_timeout 10s;

        access_log /dev/stdout;
        error_log /dev/stderr debug;
    }

    location /api/ {
        proxy_pass https://api.com;
        proxy_set_header Host api.com;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization $http_authorization;

        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_connect_timeout 10s;
        proxy_send_timeout 10s;
        proxy_read_timeout 10s;
        send_timeout 10s;

        access_log /dev/stdout;
        error_log /dev/stderr debug;
    }
}

My problem is that the first deployment works fine, but after a few days, Nginx starts behaving strangely. The routing, which is supposed to point to api.com, unexpectedly redirects to random unknown hostnames. This issue happens frequently. Every time I restart the ECS container, it works fine for a while, but then it starts pointing to another random hostname again.

Has anyone ever experience this?

multiple response that i got first

{
  "Message": "User: anonymous is not authorized to perform: es:ESHttpGet because no resource-based policy allows the es:ESHttpGet action"
}

second

404 Not Found
nginx

third enter image description here


Solution

  • Turns out I needed to add a DNS resolver to AWS DNS under the server block:

    server {
        resolver 169.254.169.253 valid=10s;
    }
    

    After that, the issue never happened again.