google-cloud-storagegsutilgoogle-cloud-sdk

How to download multiple files from Google Cloud Storage using the provided gsutil commands


I'm trying to download a batch of 10 files from Google Cloud Storage. I have a bucket that contains thousands of files. I can search and filter using the user interface until I have the 10 files selected that I need to download. I know I can manually click "download" for each one and then save to my computer, but when I select them all to download, I'm provided with the following snippet of code to copy and paste:

gsutil -m cp \
  "gs://class-reports/ShamaliChauhan-2022-06-24 20H50M33S480MS.pdf" \
  "gs://class-reports/ShamaliChauhan-2022-06-29 17H19M51S157MS.pdf" \
  "gs://class-reports/ShamaliChauhan-2022-06-29 17H21M37S096MS.pdf" \
  "gs://class-reports/ShamaliChauhan-2022-06-29 17H27M28S043MS.pdf" \
  "gs://class-reports/ShamaliChauhan-2022-06-29 17H44M49S566MS.pdf" \
  "gs://class-reports/ShamaliChauhan-2022-06-29 17H46M12S639MS.pdf" \
  "gs://class-reports/ShamaliChauhan-2022-08-19 12H01M59S639MS.pdf" \
  "gs://class-reports/ShamaliChauhan-2022-08-22 10H02M07S922MS.pdf" \
  "gs://class-reports/ShamaliChauhan-2022-09-07 11H28M15S384MS.pdf" \
  "gs://class-reports/VP1 basic-ShamaliChauhan-2022-06-24 18H07M50S806MS.pdf" \
  .

Easy peasy right? I just copy and paste this code into my Google Cloud SDK Shell (which is connected and runs gsutil right?) and I get this:

C:\Users\shaed\Desktop\Shamali>gsutil -m cp \
CommandException: Wrong number of arguments for "cp" command.

C:\Users\shaed\Desktop\Shamali>  "gs://class-reports/ShamaliChauhan-2022-06-24 20H50M33S480MS.pdf" \
The filename, directory name, or volume label syntax is incorrect.

but the 2nd error appears 10 times, once for each file I'm trying to download.

I found this question here: How to download multiple files in Google Cloud Storage using gsutil

Which mentions this same problem, and hints that perhaps it has something to do with the slashes not escaping the new-line characters (and potentially extra spaces) and so it runs each line of the suggested code separately. I'm wondering if there's something wrong with my setup, or what I need to do to be able to use the pre-made code to download multiple specified files (not all of them in a bucket) to a folder on my computer. It seems like this should be a no-brainer, but it's been a huge pain in the neck to implement.

Any ideas?


Solution

  • I know this is an old question, but since i bumped into the same issue, i'm posting my answer on this in case anyone else bumps into it as well.

    If you are trying to do this in windows, the backslash is not recognized as a valid multi-line character.

    If using CMD Try replacing \ with ^ (as @DazWilkin already pointed out).

    i.e CMD:

    gsutil -m cp ^
      "gs://class-reports/ShamaliChauhan-2022-06-24 20H50M33S480MS.pdf" ^
      "gs://class-reports/ShamaliChauhan-2022-06-29 17H19M51S157MS.pdf" ^
      "gs://class-reports/ShamaliChauhan-2022-06-29 17H21M37S096MS.pdf" ^
      "gs://class-reports/ShamaliChauhan-2022-06-29 17H27M28S043MS.pdf" ^
      "gs://class-reports/ShamaliChauhan-2022-06-29 17H44M49S566MS.pdf" ^
      "gs://class-reports/ShamaliChauhan-2022-06-29 17H46M12S639MS.pdf" ^
      "gs://class-reports/ShamaliChauhan-2022-08-19 12H01M59S639MS.pdf" ^
      "gs://class-reports/ShamaliChauhan-2022-08-22 10H02M07S922MS.pdf" ^
      "gs://class-reports/ShamaliChauhan-2022-09-07 11H28M15S384MS.pdf" ^
      "gs://class-reports/VP1 basic-ShamaliChauhan-2022-06-24 18H07M50S806MS.pdf" ^
      .
    

    Alternatively (if you would like to just use the commands as is in Windows), you could install WSL and enable linux subkernel in windows. Then you could login into the linux subkernel from the terminal ( by typing wsl in the windows terminal), install the relevant required utilis for linux, and run the commands from inside the WSL subsystem