pythonamazon-s3boto3cloud-storage

Read file from S3 into Python memory

I have a large csv file stored in S3, I would like to download, edit and reupload this file without it ever touching my hard drive, i.e. read it straight into memory from S3. I am using the python library boto3, is this possible?

Solution

  • You should look into the io module

    Depending on how you want to read the file, you can create a StringIO() or BytesIO() object and download your file to this stream.

    You should check out these answers:

    1. How to read image file from S3 bucket directly into memory?
    2. How to read a csv file from an s3 bucket using Pandas in Python