I have a github workflow which is supposed to login via SSH into a remote server and then download the docker-compose file from a private github repository.
I was first struggling to get the multi-line string syntax right for the run section in the workflow yaml file, in particular the curl command, since it requires quotes for the header. The solution i found is to split the curl command into environment variables.
download-file:
name: download file
runs-on: ubuntu-latest
steps:
- name: install ssh keys
# check this thread to understand why its needed:
# <https://stackoverflow.com/a/70447517>
run: |
install -m 600 -D /dev/null ~/.ssh/id_rsa
echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
host='${{ secrets.SSH_HOST }}'
hosts="$(dig +short "$host" | grep -v '\.$' | sed -z 's|\n|,|g')$host"
ssh-keyscan -H "$hosts" > ~/.ssh/known_hosts
- name: download docker compose file from private repo to server
run: ssh ${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }}
# the first curl call to the public repo works.
# the file can be opened. the second curl call also
# finishes but the file can NOT be opened.
"ls -la
&& cd test
&& ls -la
&& curl curl $CURL_PUBLIC_URL -o $URL_PUBLIC_O
&& curl -v -H $CURL_H1 -H $CURL_H2 -H $CURL_H3 -o $CURL_O $CURL_URL
&& exit"
env:
CURL_H1: '"\"Authorization: Bearer ${{secrets.PULL_ACCESS_TOKEN}}\""'
CURL_H2: '"\"Accept: application/vnd.github.v3.raw\""'
CURL_H3: '"\"Content-Type: text/html; charset=UTF-8\""'
CURL_O: "docker-compose.yml"
CURL_URL: "https://raw.githubusercontent.com/<username>/<repository name>/refs/heads/main/docker-compose.prod.yml"
CURL_PUBLIC_URL: "https://raw.githubusercontent.com/thnk2wn/rasp-cat-siren/master/siren/Dockerfile"
CURL_PUBLIC_O: "Dockerfile"
- name: cleanup
run: rm -rf ~/.ssh
PROBLEM:
When I execute the curl command after logging in manually into the server, the file is downloaded and I can open it, no problem.
Within the github workflow, the curl request also executes without any error, but when I try to open the downloaded file docker-compose.yml
via cat docker-compose.yml
after ssh manually into the server, the file can not be opened and the message found an invalid character in header name
is displayed instead.
My guess is that there is some kind of issue with the string formatting of the variables. It already took me hours experimenting with various string formats escaping the double quote with the backslash to keep the double quote. The string syntax looks already quite overly complicated to me to be honest.
EDIT 1: When I use wget
instead of curl in the actions run command, then i get a 400 response. When I execute the wget request manually on the server, the file is downloaded without any issue.
EDIT 2: Downloading a file from a public github repository works without any problems and can be opened with no issues.
QUESTION:
How can I run the curl request after ssh into the server downloading the single docker-compose file from the private github repo without corrupting the file so that the file can actually be opened?
Respectively, is there an issue with the formatting of the syntax for the curl command within run
considering the string formatting of the env
variables and if so, what would be a better/ correct way?
Thanks for your help!
git example, maybe need some minor to adjust env names
cat $sshKey > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
# ssh-keyscan -t rsa,dsa github.com >> ~/.ssh/known_hosts
git archive --format=tar --remote=git@YOUR-REPO.git "+YOUR-BRANCH+" docker-compose.prod.yml | tar -O -xf - > docker-compose.yml
Note: I don't know what you'll do next. But it's never a good idea to use docker-compose in prod. It's just to start, you need to avoid it when your project grow up