dockeramazon-ec2portrasa

Opening a port to the world on AWS not working


I am having trouble accessing a service that is running in a docker container (port 5005) from the internet over TCP.

The server is a ubuntu AWS ec2 instance with port 5005 open in the security group (both v4 and v6 addressing)

AWS Inbound Security Settings

The docker processes are running fine, appearing to map the port from inside its container to the ec2 instance.

ubuntu@ip-172-31-5-89:~$ docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS                                       NAMES
71e620ea2969   rasa/rasa-sdk:latest    "./entrypoint.sh sta…"   15 minutes ago   Up 15 minutes   0.0.0.0:5055->5055/tcp, :::5055->5055/tcp   emma_action_server_1
533010182ca7   rasa/rasa:latest-full   "rasa run --enable-a…"   15 minutes ago   Up 15 minutes   0.0.0.0:5005->5005/tcp, :::5005->5005/tcp   emma_rasa_1

(yes, 5005 and 5055 are both valid ports and not a typo - but only 5005 should be exposed to the ec2 instance and up through the firewall out to the web. ufw appears to be signalling the port fine.

Status: active

To                         Action      From
--                         ------      ----
5005/tcp                   ALLOW       Anywhere                  
5005                       ALLOW       Anywhere                  
22                         ALLOW       Anywhere                  
5005/tcp (v6)              ALLOW       Anywhere (v6)             
5005 (v6)                  ALLOW       Anywhere (v6)             
22 (v6)                    ALLOW       Anywhere (v6)      

and the ec2 instance appears to be listening fine:

ubuntu@ip-172-31-5-89:~$ sudo netstat -plunta | grep LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      561/systemd-resolve 
tcp        0      0 0.0.0.0:5055            0.0.0.0:*               LISTEN      6473/docker-proxy   
tcp        0      0 0.0.0.0:5005            0.0.0.0:*               LISTEN      6451/docker-proxy   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      810/sshd: /usr/sbin 
tcp6       0      0 :::5055                 :::*                    LISTEN      6480/docker-proxy   
tcp6       0      0 :::5005                 :::*                    LISTEN      6458/docker-proxy   
tcp6       0      0 :::22                   :::*                    LISTEN      810/sshd: /usr/sbin 

Yet, when I try accessing public.IP.address:5005 on any online port checking tool - it says the port is closed. When I actually try to make a POST request via postman - I get ETIMEDOUT which Im not sure is another way to say its closed, or infact, its just a timeout... but when I make the same POST request on the server, using local addressing, it works fine.

This works locally on ec2 (outside of container):

curl -XPOST localhost:5005/webhooks/rest/webhook -d '{"message":"hi"}'

this doesnt work - ETIMEOUT:

curl -XPOST publicIPAddressHere:5005/webhooks/rest/webhook -d '{"message":"hi"}'

The ACL and Network appear to be setup correctly also.

ACL1

ACL2

When I run the reachability analyser, it works - but thats obviously coming from inside the network from the private IP address... 172... so the issue is clearly exposing the port to the world.


Solution

  • I was able to get this working by creating a fresh ec2 instance on its own VPC/ACL with the same configuration as above.

    Not really an answer as it is a work-around - gremlins in the system.