pythonpicklejoblib

joblib: Can not load dumped model from another subfolder of parent directory


I have train a small model and use joblib to save it.
But i get an error while trying to load it.
This is my structure of directory:

loader.py
package
    model_folder
        __init__.py
        model.py
        model.pkl
    predict_folder
        __init__.py
        predict.py

Inside the package.predict_folder.predict.py, i write a funtion to load the model.pkl.

from ..model import model
import joblib
import sys

def predict(model_path):
    # sys.path.append(model_path)
    model = joblib.load(model_path)

Then inside the loader.py, I call that function.

from package.predict_folder.predict import predict
predict('package/model_folder/model.pkl')

But i get this error: ModuleNotFoundError: No module named 'model_folder'

I have tried to use sys to add path: sys.path.append('package/model_folder/model.pkl') like i commented it above. But nothing has changed. Can anyone help me ?


Solution

  • I got this error ModuleNotFoundError: No module named 'model_folder' when i reconstruct my folder structure.
    The previous structure is:

    train.py
    model_folder
        model.pkl
    

    After training, i directly save the model by joblib.dumb(model, 'model_folder/model.pkl').
    Then i reconstruct:

    predict.py
    package
        model
            model.pkl
        train
            train.py
    

    The diffrence between us is that you load your model inside package.predict_folder.predict.py, mean inside the package folder, but i load from outside package, in predict.py, like above.
    But the same error occurs for me.

    Solution is that if you reconstruct like me, you must retrain your model and resave it to model_folder, but with new path, like package/model_folder/model.pkl.

    The reason of error may because when you save model before reconstructing, joblib has remembered full path to model.pkl.
    So when you load it, joblib just use the old path, model_folder/model.pkl, instead the new one, package/model_folder/model.pkl.