bashamazon-ec2sshaws-ssm

SSH ProxyCommand using aws SSM session manager and bash script with select command


In my company when we SSH to our AWS EC2 instances we are required to use the aws CLI session-manager plugin for auth. Using this SSH config snippet works:

Host my-aws-host
    ProxyCommand bash -c "aws ssm start-session --target 'i-0abc123def456hij' \
        --document-name AWS-StartSSHSession --parameters 'portNumber=22' \
        --region us-west-1 --profile MAIN"

However, when the EC2 instance is relaunched, which happens semi-regularly, the 'target' instance ID changes. When this happens, all users need to update their SSH config with the new ID. We don't have any sort of DNS that resolves these instances to a static hostname unfortunately, and so would need to somehow publish the new instance ID to all interested users.

So instead I wrote a bash script (ssh_proxy_command.sh) that first queries our AWS account to grab the current instance ID based on a known tag value, and use that for the target - here's a cut-down version:

#!/bin/bash

INSTANCE_ID=$(aws ec2 describe-instances --region us-west-1 \
        --filters Name=tag:Name,Values=my-server-nametag* \
        --query "Reservations[*].Instances[*].{Instance:InstanceId}" --output text)

aws ssm start-session --target $INSTANCE_ID --document-name AWS-StartSSHSession --parameters 'portNumber=22' --region us-west-1 --profile MAIN

Now the SSH config looks like

Host my-aws-host
    ProxyCommand bash -c "/path/to/my/ssh_proxy_command.sh %h"

This has been working fine. However, we have just started running multiple instances built from the same base image (AMI), and which use the same tags, etc. so the given describe-instances query now returns multiple instance IDs. So I tried wrapping the output returned by the query in a bash select loop, thinking I could offer the user a list of instance IDs and let them choose the one they want. This works when running the script directly, but not when it's used as the ProxyCommand. In the latter case when it reaches the select statement it prints out the options as expected, but doesn't wait for the user input - it just continues straight to the end of the script with an empty $INSTANCE_ID variable, which makes the aws ssm command fail.

I'm guessing this is a side-effect of the way SSH runs its ProxyCommands — from the ssh_config man page:

[the proxy command] is executed using the user's shell ‘exec’ directive [...]

I'm hoping I can find a way around this problem while still using SSH config and ProxyCommand, rather than resorting to a complete stand-alone wrapper around the ssh executable and requiring everyone use that. Any suggestions gratefully accepted...


Solution

  • I faced the same issue trying to achieve exactly what you're trying to do here.

    I solved by writing and reading to/from tty as it seems the only stream available in the spawned exec command is /dev/stderr (select command outputs to stderr). Following, the solution I adopted:

    ~/.ssh/config:

    ProxyCommand sh ~/ssm-node-selection.sh <PROFILE>
    

    ~/ssm-node-selection.sh:

    PROFILE=$1
    TTY="/dev/tty$(ps ax | grep "^$$" | awk '{ print $2 }')"
    
    EC2_INSTANCE_IDS=$(aws ec2 describe-instances --profile $PROFILE --query "Reservations[].Instances[].InstanceId" --filters "Name=tag:Name,Values=<INSTANCE_NAME>" | jq '. | join(" ")' | tr -d '"')
    echo $EC2_INSTANCE_IDS > $TTY
    echo "Choose an instance: " > $TTY
    read INSTANCE_ID < $TTY
    
    aws ec2-instance-connect send-ssh-public-key --profile $PROFILE --instance-id $INSTANCE_ID --instance-os-user <USER> --ssh-public-key file://<PUBLIC_KEY> && aws ssm start-session --target $INSTANCE_ID  --profile $PROFILE --document-name AWS-StartSSHSession