[SOLVED] Server Side Encryption in to

Server Side Encryption in to_csv function

I'm getting this error while using to_csv("s3://mys3bucket/result.csv")

Exception: [Errno Write Failed: mys3bucket/result.csv/2489.part]
An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

It may have been caused because Dask is not using server-side encryption. Please tell me how can I make it use SSE or some other method to successfully write the file to the s3 bucket

Solution

Expanding on @user32185's comment, for controling SSE via dask, your call should look something like this:

to_csv("s3://mys3bucket/result.csv",
       storage_options={'s3_additional_kwargs':
           {'ServerSideEncryption': 'AES256'}})

where the specifics of SSE with s3fs are detailed here. Note that you may also require other keywords from the same docs page, for credentials, storage zone, etc. The parameters are passed to the S3FileSystem constructor, and you can delve into the boto docs to see what everything means.