dockermacospsqlportforwardingaws-ssm

Error connecting to psql database through tunnel on docker port


The Issue

I'm running an aws ssm port forwarding session inside a docker container which establishes a connection to a remote postgresql db. The container port maps to my localhost on port 5432. When I attempt to connect with the psql client, I get the following error:

psql: error: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

Problem Setup and Debugging:

My local machine is mac m1. Running psql client 14.11.

[local machine] Ensure no process is running (to eliminate possibility of port conflict):

netstat -an | grep LISTEN | grep 5432
echo $?
> 1

[container] Docker container is running as follows:

docker run --rm -it -p 5432:5432 \
--entrypoint=/bin/bash \
-v ~/.aws:/home/connect/.aws \
<image_name:version>

### The docker image i'm running is a debian-based 
### ruby image which has some added gems and psql and mysql clients 

# I then start an aws ssm session
aws ssm start-session --profile <PROFILE> \
--region <REGION> --target <INSTANCE_ID> \
--document-name AWS-StartPortForwardingSessionToRemoteHost \
--parameters host=<HOSTNAME>,portNumber=5432,localPortNumber=5432

[local machine]Then I ensure that the container is listening on the port:

netstat -an | grep LISTEN | grep 5432
>tcp46      0      0  *.5432                 *.*                    LISTEN

Try to establish a connection:

psql -h 127.0.0.1 -p 5432 -U <USER> -d <DB_NAME> -W
Password:
> psql: error: connection to server at "127.0.0.1", port 5432 failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

Other things I've tried:

Conclusion

I'm suspecting an issue/bug with psql mac client or docker. If anyone can confirm they're getting the same behavior or have ideas on how I can further debug/narrow down this issue that would be of great help!


Update

Looks like it's a bug in ssm. It behaves differently inside a container and does not trigger a listener on the port: https://github.com/aws/session-manager-plugin/issues/14


Solution

  • Turns out this is a bug in ssm that prevents a listener from listening on a docker port: https://github.com/aws/session-manager-plugin/issues/14

    The workaround is to change the following line in /etc/hosts

    127.0.0.1   localhost
    

    to

    0.0.0.0     localhost