Libcrypto error in Docker container in Azure DevOps pipeline

I have a Azure DevOps pipeline that I use for Terragrunt CI. The source Terraform modules that I am using in my repo are stored in a second repo, and so in order to run terragrunt plan you need to have an SSH key. The pipeline runs all Terragrunt commands in a Docker container, and I add the SSH keys to said container when it gets built. This is my Dockerfile:

FROM devopsinfra/docker-terragrunt:azure-tf-1.5.5-tg-0.50.1

RUN apt-get update
RUN apt-get install -y git

ARG SSH_PRIVATE_KEY

RUN mkdir /root/.ssh/
RUN echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa

RUN touch /root/.ssh/known_hosts
RUN touch /root/.ssh/config
RUN ssh-keyscan -t rsa ssh.dev.azure.com >> /root/.ssh/known_hosts
RUN echo $'Host ssh.dev.azure.com vs-ssh.visualstudio.com \n\
           HostkeyAlgorithms +ssh-rsa \n\
           IdentityFile ~/.ssh/id_rsa \n\
           IdentitiesOnly yes' >> /root/.ssh/config

RUN chmod -R 600 ~/.ssh

This is the relevant part of my ADO pipeline:

- task: Docker@1
  displayName: 'Building image'
  inputs:
    command: 'Build an image'
    imageName: 'tg-azure-img'
    arguments: '--build-arg SSH_PRIVATE_KEY="$(ado-privatekey)"'
- task: Docker@1
  displayName: 'Terragrunt Plan'
  inputs:
     command: 'Run an image'
     imageName: 'tg-azure-img'
     qualifyImageName: false
     volumes: '$(Pipeline.Workspace)/drop:/code'
     envVars: |
       ARM_CLIENT_SECRET=$(edsp-terraform-ci-credential)
       ARM_CLIENT_ID=$(arm-client-id)
       ARM_TENANT_ID=$(arm-tenant-id)
       ARM_SUBSCRIPTION_ID=$(arm-subscription-id)
     workingDirectory: '/code/subscriptions/${{ parameters.workingDirectory }}'
     containerCommand: '/bin/bash run-plan.sh'
     runInBackground: false

The actual content of the private key are coming from a secure variable group that is referenced at the top of the pipeline. When I run the pipeline, it works fine until it gets to the part where it runs terragrunt plan, at which point I get the following error:

│ Error: Failed to download module
│ 
│   on main.tf line 1:
│    1: module "blob-storage-account" {
│ 
│ Could not download module "blob-storage-account" (main.tf:1) source code
│ from
│ "git::ssh://git@ssh.dev.azure.com/v3/myorg/modules:
│ error downloading
│ 'ssh://git@ssh.dev.azure.com/v3/myorg/modules:
│ /usr/bin/git exited with 128: Cloning into
│ '.terraform/modules/blob-storage-account'...
│ Load key "/root/.ssh/id_rsa": error in libcrypto
│ Permission denied, please try again.
│ Permission denied, please try again.
│ git@ssh.dev.azure.com: Permission denied (password,publickey).
│ fatal: Could not read from remote repository.
│ 
│ Please make sure you have the correct access rights
│ and the repository exists.
│ 
╵

I have only been able to find people having similar issues with Github pipelines, but none specific to Azure DevOps. The Github solutions didn't work for me where they were applicable. I've tried numerous different permutations of chmod commands, I've tried a bunch of things with the private key format, using the ssh-agent to manage things, but nothing has worked. Any ideas?

Load key "/root/.ssh/id_rsa": error in libcrypto

I can reproduce the same issue when using the similar dockerfile to configure ssh.

The cause of the issue could be that when you using Pipeline variable to pass the ssh private key to docker image(RUN echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa), the key shows as one-line value. The Private Key format is invalid.

To solve this issue, you can save the private key file(id_rsa) to secure files(Pipelines -> Libraries -> secure files). Then you can download the id_rsa file in Pipeline and COPY/ADD to the docker image.

Here are the steps:

Step1: Add the id_rsa file to secure files.

Step2: Modify the dockerfile to use ADD command to add the id_rsa file to docker image.

For example: ADD id_rsa /root/.ssh/id_rsa

FROM devopsinfra/docker-terragrunt:azure-tf-1.5.5-tg-0.50.1

RUN apt-get update
RUN apt-get install -y git


RUN mkdir /root/.ssh/
ADD id_rsa /root/.ssh/id_rsa
RUN touch /root/.ssh/known_hosts
RUN touch /root/.ssh/config
RUN ssh-keyscan -t rsa ssh.dev.azure.com >> /root/.ssh/known_hosts
RUN echo $'Host ssh.dev.azure.com vs-ssh.visualstudio.com \n\
           HostkeyAlgorithms +ssh-rsa \n\
           IdentityFile ~/.ssh/id_rsa \n\
           IdentitiesOnly yes' >> /root/.ssh/config

RUN chmod -R 600 ~/.ssh

Step3: In Azure Pipeline, you can use the Download secure file task to download the id_rsa file and use Copy files Task to copy the id_rsa file to the SAME PATH as dockerfile.

Here is the YAML sample:

steps:
- task: DownloadSecureFile@1
  displayName: 'Download secure file'
  inputs:
    secureFile: 'id_rsa'

- task: CopyFiles@2
  displayName: 'Copy Files to: $(build.sourcesdirectory)'
  inputs:
    SourceFolder: '$(Agent.TempDirectory)'
    Contents: 'id_rsa'
    TargetFolder: '$(build.sourcesdirectory)/samepathasdockerfile'

- task: Docker@1
  displayName: 'Building image'
  inputs:
    command: 'Build an image'
    imageName: 'tg-azure-img'

- task: Docker@1
  displayName: 'Terragrunt Plan'
  inputs:
     command: 'Run an image'
     imageName: 'tg-azure-img'
     qualifyImageName: false
     volumes: '$(Pipeline.Workspace)/drop:/code'
     envVars: |
       ARM_CLIENT_SECRET=$(edsp-terraform-ci-credential)
       ARM_CLIENT_ID=$(arm-client-id)
       ARM_TENANT_ID=$(arm-tenant-id)
       ARM_SUBSCRIPTION_ID=$(arm-subscription-id)
     workingDirectory: '/code/subscriptions/${{ parameters.workingDirectory }}'
     containerCommand: '/bin/bash run-plan.sh'
     runInBackground: false

In this case, the id_rsa file will be passed with the correct format. And the Pipeline can work as expected.

Result: