amazon-web-servicesdockernginxamazon-ecsjwilder-nginx-proxy

AWS ECS jwilder/nginx-proxy fails to generate servers inside upstream


I am trying to setup jwilder/nginx-proxy as reverse-proxy to proxy requests to various containers that expose the VIRTUAL_HOST=example.com environment variables.

The setup works if the container is started directly on the ec2 cluster host but fails with the following error: "error running notify command: nginx -s reload, exit status 1" if it is spawned from ECS.

The docker log of the container running the jwilder/nginx-proxy container: WARNING: /etc/nginx/dhparam/dhparam.pem was not found. A pre-generated dhparam.pem will be used for now while a new one is being generated in the background. Once the new dhparam.pem is in place, nginx will be reloaded. forego | starting dockergen.1 on port 5000 forego | starting nginx.1 on port 5100 dockergen.1 | 2018/08/19 10:43:37 Generated '/etc/nginx/conf.d/default.conf' from 4 containers dockergen.1 | 2018/08/19 10:43:37 Running 'nginx -s reload' dockergen.1 | 2018/08/19 10:43:37 **Error running notify command: nginx -s reload, exit status 1** dockergen.1 | 2018/08/19 10:43:37 Watching docker events dockergen.1 | 2018/08/19 10:43:37 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload' 2018/08/19 10:48:23 [emerg] 38#38: no servers are inside upstream in /etc/nginx/conf.d/default.conf:55 nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/default.conf:55 Generating DH parameters, 2048 bit long safe prime, generator 2 This is going to take a long time dhparam generation complete, reloading nginx

The environment is configured as the following:

services: 
- name: proxy
  *volumes*:
  Name: docker-socket
  Source Path: /var/run/docker.sock
  *containers*: 
    - name: proxy
      image: jwilder/nginx-proxy
      port: 80:80
      Mount Points:
        Container Path: /tmp/docker.sock
        Source Volume: docker-socket
        Read only: true 
- name: site
  *containers*:
    - name: site
      image: nginx
      port: 0:80
      environment:
      - VIRTUAL_HOST=example.com

Command to test:

curl -H "Host: example.com" localhost:80   

It now returns the default nginx page because the nginx config file failed to generate a valid config because the upstream hosts are missing.

The generated invalid nginx config

proxy_set_header Proxy "";
server {
  server_name _; # This is just an invalid value which will never trigger on a real hostname.
  listen 80;
  access_log /var/log/nginx/access.log vhost;
  return 503;
}
# example.com
upstream example.com {
}
server {
  server_name example.com;
  listen 80 ;
  access_log /var/log/nginx/access.log vhost;
  location / {
    proxy_pass http://example.com
  }
}

The proxy works as intended if the following command is used:

docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy

If the command above is run it gives the following output: WARNING: /etc/nginx/dhparam/dhparam.pem was not found. A pre-generated dhparam.pem will be used for now while a new one is being generated in the background. Once the new dhparam.pem is in place, nginx will be reloaded. forego | starting dockergen.1 on port 5000 forego | starting nginx.1 on port 5100 dockergen.1 | 2018/08/19 10:18:48 Generated '/etc/nginx/conf.d/default.conf' from 10 containers dockergen.1 | 2018/08/19 10:18:48 Running 'nginx -s reload' dockergen.1 | 2018/08/19 10:18:48 Watching docker events dockergen.1 | 2018/08/19 10:18:48 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload' 2018/08/19 10:19:09 [notice] 40#40: signal process started Generating DH parameters, 2048 bit long safe prime, generator 2 This is going to take a long time dhparam generation complete, reloading nginx The generated valid nginx config:

proxy_set_header Proxy "";
server {
  server_name _; # This is just an invalid value which will never trigger on a real hostname.
  listen 80;
  access_log /var/log/nginx/access.log vhost;
  return 503;
}
# example.com
upstream example.com {
        ## Can be connected with "bridge" network
      # ecs-site-site-add8hjasd
      server 172.17.0.3:80;
}
server {
  server_name example.com;
  listen 80 ;
  access_log /var/log/nginx/access.log vhost;
  location / {
    proxy_pass http://example.com;
  }
}

My question is: why doesn't this work, is it because permissions or mount to the docker socket?


Solution

  • 3 days ago, Our team had met this question. We spent a lot of time for that.

    Problem reason should be in AWS ecs-agent(we have 2 envrionment, one ecs-agent's version is 1.21 and another one is 1.24)

    Yesterday, We solved this problem: Using AWS console to update ecs-agent to the latest version: 1.34 and restart the ecs-agent(docker contianer) Then the problem was solved.

    Just paste this solution here. Hope it would be helpful to others!