How to structure Python package to allow for testing

I'm having trouble being able to get my code to be executable and testable. Here's my project's file structure:

project/
├── .pytest_cache/
│
├── src/
│   ├── __init__.py
|   ├── .pytest_cache/
│   ├── module1.py
│   └── module2.py
│
├── tests/
|   ├── .pytest_cache/
|   ├── __init__.py
|   └── test_module.py
|
├── venv/

where both __init__.py files are empty

I'm using pytest and calling test_module.py by calling pytest in the cli at the root level of the project. This works when I use absolute imports in all of my modules in src: from src.module2.py import some_func, some_other_func
but then executing module1.py from the cli no longer works:

Traceback (most recent call last):
  File "C:\project\src\module1.py", line 3, in <module>
    from src.module2 import some_func, some_other_func,
ModuleNotFoundError: No module named 'src'

However, when I remove the src. from the import statements, I can execute the scripts just fine, but no longer test them.

I've tried appending my src directory to the project path in the testfile:

test_module.py:

import sys
import os

sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../src')))

Also tried this:

test_module.py:

import os, sys
sys.path.append(os.path.join(os.getcwd(), os.path.pardir))

then calling my test in the root of my project: python -m tests.test_module and nothing happens.

What can I do so executing tests & my scripts are possible? Are there any libs/packages to make this easier?

Solution

Before diving into testing, ensure your project is structured properly.

Project Structure

Your src/ directory should contain a subdirectory with the name of your module:

super-cool-module/
├── src/
│   └── super_cool_module/
│       ├── __init__.py
│       ├── submodule1.py
│       └── submodule2.py
├── tests/
│   ├── __init__.py
│   ├── test_submodule1.py
│   └── test_submodule2.py
├── venv/
├── setup.py    # setuptools
├── README.md   # Documentation
└── .gitignore  # Exclude venv, etc.

Alternatively, if you prefer not to use a src/ directory:

super-cool-module/
├── super_cool_module/
│   ├── __init__.py
│   ├── submodule1.py
│   └── submodule2.py
├── tests/
│   ├── __init__.py
│   ├── test_submodule1.py
│   └── test_submodule2.py
├── venv/
├── setup.py    # setuptools
├── README.md   # Documentation
└── .gitignore  # Exclude venv, etc.

Testing Setup

To run tests, you have a few options:

Using PYTHONPATH: Run pytest with the path to your src directory:
```
PYTHONPATH=src pytest
```
Pytest Configuration: Alternatively, you can set up a pytest.ini configuration file:
```
# pytest.ini
[pytest]
pythonpath = src
```
Editable Mode Installation: If you’ve activated your virtual environment, install your module in editable mode:
```
source venv/bin/activate
pip install -e .
```
Now you can run tests by simply calling:
```
pytest
```
This works because your module is installed in your local environment.

Note: Remember to exclude the venv/ directory from version control by adding it to your .gitignore file.