python-3.xserializationpickledillcloudpickle

Python3 C-libs: How do I serialize a class that depends on C .dlls for use in a bare instance?


For example:

// file foo
def my_func():
   print("foo!")

// file bar
class Bar:
   def __init__(self):
      pass 
   def hello():
      import foo.my_func
      my_func()

import dill
dill.dump(Bar(), open("bar.dpkl",mode='wb+'))
$: python3 -m venv foobar
$: source activate foobar/bin/activate
(foobar) $: python3
>>> import dill
>>> obj = dill.load(open("bar.dpkl",mode='rb')) # just fine
>>> obj.hello()
foo!

But, suppose I throw in a twist:

class Bar:
  def __init__(self):
     import numpy

Now,

>>> obj = dll.load(open("bar.dpkl",mode='rb'))
ModuleNotFoundError: No module named 'numpy'

It appears as if the linked C-binaries are not bundled up in the pickle file. However, if the code is genuine, native python code, the pickle file output by dill is essentially a dynamic archive.

Is there any way to extend this dynamic archive behavior to include the libraries like numpy et al.? Or must these sort of C + python-shell libraries always be installed in the executing python environment?


Basically, I think the thing I am asking for is a pythonic version of a jar that can be generated straight from python code: is this possible? Or is a container with requisite dependencies the jar I am looking for?


Solution

  • A possible solution:

    import dill
    
    class Bar:
      def __init__(self, foo):
        self.basename = foo.name
        self.deps = deepcopy(foo.deps)
        self.foo = '-'.join([self.basename,'foo'])
        self.bar = '-'.join([self.basename,'bar'])
    
      def install(self):
        import pip
        pip.main(["install"] + self.deps) # this method will be deprecated soon
    
      def load(self):
        self.install()
        foo = dill.load(open(self.foo,'rb'))
        return foo
     
      def store_self():
        with open(self.bar + '.dpkl','wb+') as out:
          dill.dump(out,self)
    
      @classmethod
      def store(cls, foo):
        bar = cls(foo)
        bar.store_self()
        bar.install()
        with open(self.foo + '.dpkl','wb+') as out:
          dill.dump(out,foo)
    
    class Foo:
      def __init__(self,name,*arg,**kwargs):
        self.name = name
        self.deps = [ "numpy" ]
    
      def hello():
        import numpy
        print("foo")
        ...
    
    
    foo = Foo("Hello! I'm a foo!")
    Bar.store(foo)
    

    $: python3 -m venv venv-foo
    $: source venv-foo/bin/activate
    $: python3 -m pip install dill
    >>> import dill
    >>> bar = dill.load(open('bar.dpkl','rb'))
    >>> foo = bar.load()
    >>> foo.hello()
    Hello! I am a foo!
    

    No warranty of copy-pasteability, but the gist of it is there. Might have to tinker with it a bit to get the Bar.dpkl working. It is also sensitive to python3's version.

    With some more work, native pickle can probably be used to store Bar, which can then be used to install dill.