amazon-web-servicesbashamazon-s3jqaws-cli

How to avoid "Invalid argument type" error when executing a seemingly valid AWS CLI command created using jq and bash?


I have a file of s3 paths, sources.lst. Contents:

[
  "s3://bucket-name/path/to/file0.tif",
  "s3://bucket-name/path/to/file1.tif",
]

I am attempting to iterate over these s3 paths & download the files. I am trying to using jq and a bash while loop. My download.sh:

#!/bin/bash

set -o errexit

while read -r source
do
    echo "Attempting to execute:"
    echo "aws s3 cp "${source}" ."

    aws s3 cp "${source}" .
done < <(cat sources.lst | jq '.[]')

Executing bash download.sh gives me:

Attempting to execute:
aws s3 cp "s3://bucket-name/path/to/file0.tif" .

usage: aws s3 cp <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>
Error: Invalid argument type

Which is strange, since if I copy the line causing the error, ie

aws s3 cp "s3://bucket-name/path/to/file0.tif" .

and execute it in the terminal I run bash download.sh, the download succeeds.

How do I download the files specified in sources.lst using bash?

I am using Mac, bash --version: GNU bash, version 5.2.26(1)-release (aarch64-apple-darwin22.6.0), aws --version: aws-cli/2.13.1 Python/3.11.4 Darwin/23.4.0 exe/x86_64 prompt/off


Solution

  • Use jq's --raw-output (or -r) flag to decode JSON strings into raw text (essentially resolving escaped characters and removing the double quotes). Without it, the Bash variable storing each result would still contain the encoded output. (Yet, manually copying that into a command on the terminal may accidentally still work because the shell would consume those quotes when reading the command.)

    while read -r source
    do aws s3 cp "${source}" .
    done < <(jq -r '.[]' sources.lst)