I have a file of s3 paths, sources.lst
. Contents:
[
"s3://bucket-name/path/to/file0.tif",
"s3://bucket-name/path/to/file1.tif",
]
I am attempting to iterate over these s3 paths & download the files. I am trying to using jq
and a bash while loop. My download.sh
:
#!/bin/bash
set -o errexit
while read -r source
do
echo "Attempting to execute:"
echo "aws s3 cp "${source}" ."
aws s3 cp "${source}" .
done < <(cat sources.lst | jq '.[]')
Executing bash download.sh
gives me:
Attempting to execute:
aws s3 cp "s3://bucket-name/path/to/file0.tif" .
usage: aws s3 cp <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>
Error: Invalid argument type
Which is strange, since if I copy the line causing the error, ie
aws s3 cp "s3://bucket-name/path/to/file0.tif" .
and execute it in the terminal I run bash download.sh
, the download succeeds.
How do I download the files specified in sources.lst
using bash?
I am using Mac, bash --version
: GNU bash, version 5.2.26(1)-release (aarch64-apple-darwin22.6.0)
, aws --version
: aws-cli/2.13.1 Python/3.11.4 Darwin/23.4.0 exe/x86_64 prompt/off
Use jq's --raw-output
(or -r
) flag to decode JSON strings into raw text (essentially resolving escaped characters and removing the double quotes). Without it, the Bash variable storing each result would still contain the encoded output. (Yet, manually copying that into a command on the terminal may accidentally still work because the shell would consume those quotes when reading the command.)
while read -r source
do aws s3 cp "${source}" .
done < <(jq -r '.[]' sources.lst)