pythonpython-3.10shelvepython-3.11

How to migrate `shelve` file from Python 3.10 to Python 3.11


Code:

with shelve.open("cache") as db:
   ...

Python 3.10.9 result

A cache.db file is created.

Python 3.11.1 result

Three files are created: cache.cir, cache.bak, cache.dat.

What I need

I have important data in the old file and I need to keep using that data after updating from Python 3.10.9 to Python 3.11.1. How can I migrate the file to a new format?

What I tried

I was looking at the shelve documentation and at the Python changelog. There is no information.


Solution

  • A shelf is just a key-value database of pickle data – I suppose your Python 3.11 build doesn't have the bsddb module so it's falling back to a dbm file.

    Write the entire shelf into a single pickle file in Python 3.10, then read it back and write into a new shelf in Python 3.11. (You can't efficiently random-access a pickle file by key, hence it's only used temporarily here.)

    You can run this same script with the two different interpreters.

    import shelve
    import pickle
    import sys
    
    pickle_name = "cache.pickle"
    old_shelf = "cache"
    new_shelf = "cache-new"
    
    if sys.version_info < (3, 11):
        print("Porting shelf to pickle")
        with shelve.open(old_shelf) as db, open(pickle_name, "wb") as f:
            pickle.dump(dict(db), f)
    else:
        print("Porting pickle to shelf")
        with open(pickle_name, "rb") as f, shelve.open(new_shelf) as db:
            db.update(pickle.load(f))