pythonpython-importpython-wheelprotobuf-python

Implicit relative imports inside of Protobuf-generated Python package make importing from outside impossible


I have a package that looks like this:

setup.py
requirements.txt
alphausblue/
    api/
        ripple/
            org_pb2.py
    org/
        v1/
            org_pb2_grpc.py

In org_pb2_grpc.py I have the following import line:

from api.ripple import org_pb2 as api_dot_ripple_dot_org__pb2

The problem I'm having is that, when I import from inside the alphausblue directory, the import works correctly. However, when I create a wheel from the package, upload it to Test PyPI, download it again and attempt to import it into a test environment like this:

>>> import alphausblue
>>> import alphausblue.org
>>> import alphausblue.org.v1
>>> import alphausblue.org.v1.org_pb2_grpc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "\anaconda3\envs\test\lib\site-packages\alphausblue\org\v1\org_pb2_grpc.py", line 5, in <module>
    from api.ripple import org_pb2 as api_dot_ripple_dot_org__pb2
ModuleNotFoundError: No module named 'api'

I get the error above. I can move the code from alphausblue/ to / and then the import works but I have to do org.v1.org_pb2_grpc, which is not what I want. It seems that this should work so what am I missing here?

Update This appears to be an issue with how protoc generates the Python code. I will investigate potential solutions here and see if I can't solve the issue myself.


Solution

  • So, I tried a number of things in an attempt to get Protobuf to do what I want, i.e.. work, but was unable to figure it out. So, I made a modification to the build script to do this:

    mkdir -p generated/py/alphausblue
    python3 -m grpc_tools.protoc -I . --python_out=./generated/py/alphausblue --grpc_python_out=./generated/py/alphausblue \
            ./org/v1/*.proto \
            ./kvstore/v1/*.proto \
            ./iam/v1/*.proto \
            ./admin/v1/*.proto \
            ./cost/v1/*.proto \
            ./billing/v1/*.proto \
            ./operations/v1/*.proto \
            ./preferences/v1/*.proto
    
    python3 -m grpc_tools.protoc -I . --python_out=./generated/py/alphausblue \
            $(for v in $(find ./api -type d); do echo -n "$v/*.proto "; done)
    
    for package in $(find generated/py/alphausblue -mindepth 1 -maxdepth 1 -type d -printf "%f "); do
            echo "Found package ${package}. Beginning replacement"
            find generated/py/alphausblue/. -name '*.py' -exec sed -i -e "s/from ${package}/from alphausblue.${package}/g" {} \;
    done
    

    The first two commands here generates the Python code from the .proto files. The next command uses find and sed to get all the top-level package names in the package itself and replace the implicit relative imports with absolute imports.