curlyamlgithub-actions

github actions workflow - ssh into server, download single file via curl, unable to open file


I have a github workflow which is supposed to login via SSH into a remote server and then download the docker-compose file from a private github repository.

I was first struggling to get the multi-line string syntax right for the run section in the workflow yaml file, in particular the curl command, since it requires quotes for the header. The solution i found is to split the curl command into environment variables.

download-file:
    name: download file
    runs-on: ubuntu-latest

    steps:
      - name: install ssh keys
        # check this thread to understand why its needed:
        # <https://stackoverflow.com/a/70447517>
        run: |
          install -m 600 -D /dev/null ~/.ssh/id_rsa
          echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          host='${{ secrets.SSH_HOST }}'
          hosts="$(dig +short "$host" | grep -v '\.$' | sed -z 's|\n|,|g')$host"
          ssh-keyscan -H "$hosts" > ~/.ssh/known_hosts

      - name: download docker compose file from private repo to server
        run: ssh ${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }}
          # the first curl call to the public repo works. 
          # the file can be opened. the second curl call also 
          # finishes but the file can NOT be opened. 
          "ls -la
          && cd test
          && ls -la
         
          && curl curl $CURL_PUBLIC_URL -o $URL_PUBLIC_O
          && curl -v -H $CURL_H1 -H $CURL_H2 -H $CURL_H3 -o $CURL_O $CURL_URL
          && exit"

        env:
          CURL_H1: '"\"Authorization: Bearer ${{secrets.PULL_ACCESS_TOKEN}}\""'
          CURL_H2: '"\"Accept: application/vnd.github.v3.raw\""'
          CURL_H3: '"\"Content-Type: text/html; charset=UTF-8\""'
          CURL_O: "docker-compose.yml"
          CURL_URL: "https://raw.githubusercontent.com/<username>/<repository name>/refs/heads/main/docker-compose.prod.yml"
          CURL_PUBLIC_URL: "https://raw.githubusercontent.com/thnk2wn/rasp-cat-siren/master/siren/Dockerfile"
          CURL_PUBLIC_O: "Dockerfile"

      - name: cleanup
        run: rm -rf ~/.ssh

PROBLEM:

When I execute the curl command after logging in manually into the server, the file is downloaded and I can open it, no problem.

Within the github workflow, the curl request also executes without any error, but when I try to open the downloaded file docker-compose.yml via cat docker-compose.yml after ssh manually into the server, the file can not be opened and the message found an invalid character in header name is displayed instead.

My guess is that there is some kind of issue with the string formatting of the variables. It already took me hours experimenting with various string formats escaping the double quote with the backslash to keep the double quote. The string syntax looks already quite overly complicated to me to be honest.

EDIT 1: When I use wget instead of curl in the actions run command, then i get a 400 response. When I execute the wget request manually on the server, the file is downloaded without any issue.

EDIT 2: Downloading a file from a public github repository works without any problems and can be opened with no issues.

QUESTION:

How can I run the curl request after ssh into the server downloading the single docker-compose file from the private github repo without corrupting the file so that the file can actually be opened? Respectively, is there an issue with the formatting of the syntax for the curl command within runconsidering the string formatting of the envvariables and if so, what would be a better/ correct way?

Thanks for your help!


Solution

  • git example, maybe need some minor to adjust env names

    cat $sshKey > ~/.ssh/id_rsa
    chmod 600 ~/.ssh/id_rsa
    # ssh-keyscan -t rsa,dsa github.com >> ~/.ssh/known_hosts
    
    git archive --format=tar --remote=git@YOUR-REPO.git "+YOUR-BRANCH+" docker-compose.prod.yml | tar -O -xf - > docker-compose.yml
    

    Note: I don't know what you'll do next. But it's never a good idea to use docker-compose in prod. It's just to start, you need to avoid it when your project grow up