pythonpickletop2vec

Issue with Pickling in Top2Vec v1.0.34

I recently upgraded to Top2Vec version 1.0.34 and encountered an issue with pickling the model. Specifically, I'm getting following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[18], line 8
      2 model_name_out = os.path.join(
      3                 stt.DATA_PATH_RESULTS,
      4                 f"{PROJECT_NAME}.pickle")
      7 with open(model_name_out, "wb") as file:
----> 8     pickle.dump(data, file)

AttributeError: Can't pickle local object 'Loader._recreate_base_user_object.<locals>._UserObject'

Here all the information:

Python version: 3.9.12
Top2Vec version: 1.0.34

Question: Has anyone else encountered this issue? Are there any workarounds or known fixes for this problem?

Thank you for your help!

Solution

  • Looking at what model.save() does, it cleans up some things before pickling the instance to the given destination with joblib.dump, then restores them.

    You should thus be able to

    bio = io.BytesIO()
model.save(bio)
data = {"model_serialized": model, "dataset": reduced_df}
# ...
pickle.dump(data, file)

    and to restore things from such a pickled dict,

    data = pickle.load(file)
model = Top2Vec.load(io.BytesIO(data["model_serialized"])))