python python-3.x python-import setup.py pyc

ModuleNotFoundError with setup.py using a compiled pyc module

I can normally import a compiled .pyc module as well as .py, but when trying to package a simple project with setup.py, I'm getting the ModuleNotFoundError exception for the compiled .pyc module. Because this is only happening when using setup.py, otherwise is working fine, I don't know if there's something I should had to setup.py to make this work.

The project structure is currently something like this:

proj
├── FAILING.pyc
├── __init__.py
├── aux/
│   ├── __init__.py
│   └── aux.c
└── main.py

and the setup.py:

from setuptools import setup, Extension, find_packages

DISTNAME = 'proj'

INSTALL_REQUIRES = [
        'cython>=0.29.13',
        'numpy>=1.16.4'
]
PYTHON_REQUIRES = '>=3.6'

ENTRY_POINTS = {
    'console_scripts': ['proj = proj.main:main']
}

def setup_extensions(metadata):
    ext_modules = [Extension('proj.aux.aux', sources=['proj/aux/aux.c'])]
    metadata['ext_modules'] = ext_modules

def setup_package():
    metadata = dict(
        name=DISTNAME,
        version='0.1',
        package_dir={'': '.'},
        packages=find_packages(),
        entry_points=ENTRY_POINTS,
        python_requires=PYTHON_REQUIRES,
        install_requires=INSTALL_REQUIRES,
        zip_safe=False,
    )
    setup_extensions(metadata)
    setup(**metadata)


if __name__ == '__main__':
    setup_package()

The main.py:

#!/usr/bin/env python3

import proj.aux.aux as aux
import proj.FAILING

def main():
    print('Hello World')

If I just try to import FAILING.pyc on the repl everything works as expected:

>>> import FAILING
>>>

But if I first run python3 setup.py intall and then call proj I'm getting the following error:

$ proj
Traceback (most recent call last):
  File "/path/to/bin/proj", line 11, in <module>
    load_entry_point('proj==0.1', 'console_scripts', 'proj')()
  File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/path/to/lib/python3.8/site-packages/proj-0.1-py3.8-macosx-10.14-x86_64.egg/proj/main.py", line 4, in <module>
    import proj.FAILING
ModuleNotFoundError: No module named 'proj.FAILING'

I'm also running this inside a virtualenv environment, although I'm guessing this is not related to the error.

What am I doing wrong, or what would I need to change to make this work?

Solution

Here is a small demo which builds a source distribution and wheel that contains a .pyc file

note that I've removed most of the cruft from your example as the cython stuff is unrelated to your problem

set -euxo pipefail

rm -rf dist testpkg setup.py

cat > setup.py <<EOF
from setuptools import setup

setup(
    name='foo',
    version='1',
    packages=['testpkg'],
    package_data={'testpkg': ['*.pyc']},
)
EOF

mkdir testpkg
touch testpkg/__init__.py
echo 'print("hello hello world")' > testpkg/mod.py

python3 -m compileall -b testpkg/mod.py
rm testpkg/mod.py

python3 setup.py sdist bdist_wheel
tar --list -f dist/*.tar.gz
unzip -l dist/*.whl

there's a few things to note about the setup.py:

I include in packages the package that has the .pyc files -- I could have used setuptools.find_packages instead, but this was simpler
The .pyc file is included as package_data -- by default pyc files are not packaged as they're generally leftover build artifacts
I need to compile the pyc into the legacy location using the -b flag of python3 -m compileall
even a "compiled" pyc file does not obfuscate the actual code, it can be recovered using dis for example -- when you talk about "compiled" here it just means it has been transformed into the (still relatively high level) python bytecode

From either the source distribution or the wheel, you can install the package.

For example running the script:

$ bash t.sh
+ rm -rf dist testpkg setup.py
+ cat
+ mkdir testpkg
+ touch testpkg/__init__.py
+ echo 'print("hello hello world")'
+ python3 -m compileall -b testpkg/mod.py
Compiling 'testpkg/mod.py'...
+ rm testpkg/mod.py
+ python3 setup.py sdist bdist_wheel
running sdist
running egg_info
writing foo.egg-info/PKG-INFO
writing dependency_links to foo.egg-info/dependency_links.txt
writing top-level names to foo.egg-info/top_level.txt
reading manifest file 'foo.egg-info/SOURCES.txt'
writing manifest file 'foo.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

creating foo-1
creating foo-1/foo.egg-info
creating foo-1/testpkg
copying files to foo-1...
copying setup.py -> foo-1
copying foo.egg-info/PKG-INFO -> foo-1/foo.egg-info
copying foo.egg-info/SOURCES.txt -> foo-1/foo.egg-info
copying foo.egg-info/dependency_links.txt -> foo-1/foo.egg-info
copying foo.egg-info/top_level.txt -> foo-1/foo.egg-info
copying testpkg/__init__.py -> foo-1/testpkg
copying testpkg/mod.pyc -> foo-1/testpkg
Writing foo-1/setup.cfg
creating dist
Creating tar archive
removing 'foo-1' (and everything under it)
running bdist_wheel
running build
running build_py
copying testpkg/__init__.py -> build/lib/testpkg
copying testpkg/mod.pyc -> build/lib/testpkg
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/testpkg
copying build/lib/testpkg/__init__.py -> build/bdist.linux-x86_64/wheel/testpkg
copying build/lib/testpkg/mod.pyc -> build/bdist.linux-x86_64/wheel/testpkg
running install_egg_info
Copying foo.egg-info to build/bdist.linux-x86_64/wheel/foo-1-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/foo-1.dist-info/WHEEL
creating 'dist/foo-1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'testpkg/__init__.py'
adding 'testpkg/mod.pyc'
adding 'foo-1.dist-info/METADATA'
adding 'foo-1.dist-info/WHEEL'
adding 'foo-1.dist-info/top_level.txt'
adding 'foo-1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
+ tar --list -f dist/foo-1.tar.gz
foo-1/
foo-1/PKG-INFO
foo-1/foo.egg-info/
foo-1/foo.egg-info/PKG-INFO
foo-1/foo.egg-info/SOURCES.txt
foo-1/foo.egg-info/dependency_links.txt
foo-1/foo.egg-info/top_level.txt
foo-1/setup.cfg
foo-1/setup.py
foo-1/testpkg/
foo-1/testpkg/__init__.py
foo-1/testpkg/mod.pyc
+ unzip -l dist/foo-1-py3-none-any.whl
Archive:  dist/foo-1-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-02-17 22:27   testpkg/__init__.py
      136  2021-02-17 22:27   testpkg/mod.pyc
      163  2021-02-17 22:28   foo-1.dist-info/METADATA
       92  2021-02-17 22:28   foo-1.dist-info/WHEEL
        8  2021-02-17 22:27   foo-1.dist-info/top_level.txt
      408  2021-02-17 22:28   foo-1.dist-info/RECORD
---------                     -------
      807                     6 files

Afterwards, I can install this package and use it:

$ mkdir t
$ cd t
$ virtualenv venv
...
$ . venv/bin/activate
$ pip install ../dist/foo-1-py3-none-any.whl
...
$ python3 -c 'import testpkg.mod'
hello hello world