python-2.7pybind11openimageio

Is it possible to create a standalone file to import a python library created with pybind?


I hope I'm clear in my question, if not please tell me.

I am using OpenImageIO's python bindings (pybind11) for some scripts that will run on hundreds of computers. Unfortunately it took me a lot of time to install OpenImageIO and make it work with my Python2 installation. I'd like to know if there's a way to create a file/folder that I could send to other computers so they can install the Python module simply with "pip install file/folder".

Thanks ofr your help


Solution

  • Are you running the scripts on a compute cluster with a shared filesystem? If so, then there's no need to create separate installations of python for each machine. The simplest solution is to create ONE python environment in a location that is accessible by all of your machines. An easy way to create a Python environment in a non-system location is to use Miniconda. Install it to a shared (network) location, and create an environment for all of your machines to use.

    If your machines do NOT have a shared file system, then you'll need to somehow reproduce the environment on all of them independently. In this case, there's no simple way to do that with pip.**

    But if you can use conda instead, then there's a very straightforward solution. First, install everything you need into a single conda environment. Then you have a choice: You can export the list of conda packages, or simply copy the entire conda environment directory to the other machines.

    OpenImageIO is available from the conda-forge channel, a community-developed repository of conda packages. The name of the package is py-openimageio. They have stopped updating the python-2.7 version, but the old versions are still available.

    Here's how to do it.

    1. Install Miniconda-2.7
    1. Create a new environment with python 2.7, OpenImageIO, and any other packages you need:
    conda create -n jao-stuff -c conda-forge py-openimageio python=2.7
    conda activate jao-stuff
    python -c "import OpenImageIO; print('It works!')"
    
    1. Do ONE of the following:

      a. Export the list of packages in your environment:

      conda env export -n jao-stuff -f jao-stuff-packages.yaml
      

      Then, on the other machines, install Miniconda, then create the environments using the package list from the previous step:

      conda create -n jao-stuff --file jao-stuff-packages.yaml
      

      OR

      b. Just copy all of the files in the environment to the other machines, and run them directly. Conda environments are self-contained (except for a few low-level system libraries), so you can usually just copy the whole thing to another machine and run it without any further install step.

      tar czf jao-stuff.tar.gz $(conda info --prefix)/envs/jao-stuff
      

      On the other machine, unpack the tarball anywhere and just run the python executable it contains:

      tar xzf jao-stuff.tar.gz
      jao-stuff/bin/python -c "import OpenImageIO; print('It works!')"
      

    **That's because OpenImageIO is a C++ project, with several C++ dependencies, and they don't provide binaries in the wheel format. I don't blame them -- pip is not well suited to this use-case, even with wheels. Conda, on the other hand, was designed for exactly this use-case, and works perfectly for it.