[SOLVED] A conflicting conditional operation is currently in progress against this resource. (bucket already created)

A conflicting conditional operation is currently in progress against this resource. (bucket already created)

Using s3fs, I am uploading a file to the already created s3 bucket (not deleting the bucket). On execution, the following error is thrown:

[Operation Aborted]: A conflicting conditional operation is currently in progress against this resource.

However, I would just like to dump the pickle file into the already existing bucket rather than creating a bucket for every dump.

Could not find a helpful answer in this regard.

Solution

This was due to the wrapping of fsspec over s3fs which had a conflicting mk_dir argument. This was trying to create the bucket even after its existence in AWS.

Instead removed the the fsspec and directly used the s3fs module.

import s3fs
import pickle

file='abc.pkl'
s3=s3fs.S3FileSystem()
with s3.open(f's3:///{bucket_name}/{file}', 'wb') as f:
    pickle.dump('data_to_be_written', f)