I'm currently working on a Dash app to visualize large amounts of data. With scalability issues in mind, I'm trying to migrate from Pandas to the Vaex library to lazily load data and optimize recurrent scanning of the dataset (each time the user interacts with the filters, sampling thresholds, or other parameters).
Dash uses dcc.Store components to store data in JSON format which are essential to my app at the moment. So far, I was using to_json method to convert my Pandas dataframe objects. Unfortunately, I haven't found an equivalent for Vaex.
The built-in .export method doesn't support this format, as shown in error below :
raise ValueError('''Unrecognized file extension. Please use .arrow, .hdf5, .parquet, .fits, or .csv to export to the particular file format.''')
Also, running the command json.dump(my_vaex_df, 'my_df.json') returns :
TypeError: Object of type DataFrameLocal is not JSON serializable
Is there an equivalent to to_json for Vaex dataframe objects ? If not, would there be an alternative to tackle this issue?
Grateful for any feedback.
Vaex has an df.export_json() method. It is probably what you are looking for.