Using the IO tools in pandas it is possible to convert a DataFrame to an in-memory feather buffer:
import pandas as pd
from io import BytesIO
df = pd.DataFrame({'a': [1,2], 'b': [3.0,4.0]})
buf = BytesIO()
df.to_feather(buf)
However, using the same buffer to convert back to a DataFrame
pd.read_feather(buf)
Results in an error:
ArrowInvalid: Not a feather file
How can a DataFrame be convert to an in-memory feather representation and, correspondingly, back to a DataFrame?
Thank you in advance for your consideration and response.
With pandas==0.25.2 this can be accomplished in the following way:
import pandas
import io
df = pandas.DataFrame(data={'a': [1, 2], 'b': [3.0, 4.0]})
buf = io.BytesIO()
df.to_feather(buf)
output = pandas.read_feather(buf)
Then a call to output.head(2) returns:
a b
0 1 3.0
1 2 4.0
Note that you could do the same with csv files, but would require you to use StringIO instead of BytesIO
If you have a DataFrame with multiple indexes, you may see an error like
ValueError: feather does not support serializing <class 'pandas.core.indexes.base.Index'> for the index; you can .reset_index()to make the index into column(s)
In which case you need to call .reset_index() before to_feather, and call .set_index([...]) after read_feather
Last thing I would like to add, is that if you are doing something with the BytesIO, you need to seek back to 0 after writing the feather bytes. For example:
buffer = io.BytesIO()
df.reset_index(drop=False).to_feather(buffer)
buffer.seek(0)
s3_client.put_object(Body=buffer, Bucket='bucket', Key='file')