file-ioopenstack-swift

How to save output of python-swiftclient to file when dowloading a directory?


Sometimes I get errors when I download files from a cloud with python-swiftclient, like this one:

Error downloading object 'uploads/1/image.png': Object GET failed: https://orbit.brightbox.com/v1/acc-12345/uploads/1/image.png 500 Internal Error b'An error occurred'

To search for the all errors and re-download failed files I would want to save output of the swift command to a file

I tried to do the following ways:

swift-cli -A https://orbit.brightbox.com/v1/acc-12345 \
  -U user -K secret download uploads 2>&1 | tee uploads.log

# and

swift-cli -A https://orbit.brightbox.com/v1/acc-12345 \
  -U user -K secret download uploads > uploads.log

But this didn't work. man swift describes -o option

For a single object download, you may use the -o [--output] option to redirect the output to a specific file or if "-" then just redirect to stdout or with --no-download actually not to write anything to disk.

but when I try to download a directory with -o option if fails with

-o option only allowed for single file downloads

How can I save log to a file when I download a directory with swift CLI?


Solution

  • Actually redirecting output to a file works with swift-client:

    swift-cli -A https://orbit.brightbox.com/v1/acc-12345 \
      -U user -K secret download uploads > uploads.log
    

    I was confused because after I started the command above, in another terminal window I did

    tail -f uploads.log
    

    But it didn't give me any output (like I was seeing when I was running the download command without redirection).

    Seems like that swift-client writes to a file in batches and I needed to wait about a minute until tail -f dumps into the console a hundred of lines like this

    uploads/documents/1/image.png [auth 0.000s, headers 0.390s, total 14.361s, 0.034 MB/s]