amazon-web-servicesamazon-s3s3cmd

How can I list all empty folders with s3cmd


I have uploaded 5k+ folders, each of which should have one file and one subfolder (this subfolder then holds various files) to DigitalOcean Spaces (S3 storage).

It looks like some of the uploads failed (long story). Is there a way to list all empty folders using s3cmd?


Solution

  • My comments here are about Amazon S3, but should apply equally to DigitalOcean Spaces.

    'Folders' do not actually exist in S3. For example, you could upload a file to invoices/january/inv1.txt and S3 will magically create the invoices and january folders. Then, if you delete that object, those folders will magically disappear. Thus, folders automatically appear when objects are 'in' them.

    It is possible to create an empty folder by creating a zero-length object with the same name as a path. For example, creating a zero-length object with a key of invoices/ will force a folder to appear even when it is empty (because it isn't actually empty!). This is how the S3 management console creates a folder when people click the "Create Folder" button.

    So, when you ask how to "list all empty folders", it really depends on how those folders were originally created, or if they were actually created at all! It is quite likely that the folders were never created in the first place.

    If your goal is to fix a failed upload, you could use the s3cmd sync function that can re-upload objects, but is smart enough to only copy files that are not present in the destination, or have changed.