pythonpython-3.xamazon-s3feather

Save a data frame to S3 in feather format


I have a data frame, let's say:

import pandas as pd
df = pd.DataFrame({'a': [1, 4], 'b': [1, 3]})

I want to save it as a feather file to s3 but I can't find a working way to do it.

I tried to use s3bp and s3fs but they don't do the trick.

Any suggestion?


Solution

  • The solution that worked for me is

    import boto3
    import pandas as pd
    
    from io import BytesIO
    from pyarrow.feather import write_feather
    
    df = pd.DataFrame({'a': [1, 4], 'b': [1, 3]})
    
    s3_resource = boto3.resource('s3')
    with BytesIO() as f:
        write_feather(df, f)
        s3_resource.Object('bucket-name', 'file_name').put(Body=f.getvalue())