python-3.xamazon-web-servicesamazon-s3boto3python-s3fs

How to use Python's S3FS to delete a file that contains dashes and brackets in the filename?


I have a file named data_[2022-10-03:2022-10-23].csv.gzip in S3, inside a bucket and folder named s3://<bucket_name>/data/cache/

I am attempting to delete this file using S3FS. When I attempt to do so with:

filename = 'data_[2022-10-03:2022-10-23].csv.gzip'
data_filepath = 's3://<bucket_name>/data/cache'
fs.rm(f"{data_filepath}/{filename}")

But the rm call throws exception:

error('bad character range 2-1 at position 54')

It seems it is thinking this is a regex.

I attempted escaping it with:

filename = filename.replace('[', '\\[').replace(']', '\\]').replace('-', '\\-')

but I get the same error:

('bad character range \\\\-1 at position 57',)

I also tried re.escape():

fs.rm(f"{data_filepath}/{re.escape(filename)}")

But I get:

error('bad character range \\\\-1 at position 57')

The same thing.

I tried the delete method:

fs.delete(f"{data_filepath}/{filename}")

Same error.

I tried accessing the underlying boto3 and deleting directly:

bucket_name = '<bucket_name>'
path = 'data/cache'
filename = 'data_[2022-10-03:2022-10-23].csv.gzip'
fs.s3.delete_object(Bucket=bucket_name, Key=f'{path}/{filename}')

This method executes with no errors, but nothing is deleted.

How can I delete this file?


Solution

  • Tested this with boto3 as below to see it deleting the file as it is:

    import boto3
    
    def s3_delete():
        fs = boto3.client('s3', aws_access_key_id="########", aws_secret_access_key="########")
        bucket_name = '<bucket-name>'
        path = 'data/cache'
        filename = 'data_[2022-10-03:2022-10-23].csv.gzip'
    
        fs.delete_object(Bucket=bucket_name, Key=f'{path}/{filename}')
    
    
    s3_delete()