How to test a wheel against multiple versions of python?

Problem description

I'm writing a python library and I am planning to upload both sdist (.tar.gz) and wheel to PyPI. The build docs say that running

python -m build

I get sdist created from the source tree and wheel created from the sdist, which is nice since I get the sdist tested here "for free". Now I want to run tests (pytest) against the wheel with multiple python versions. What is the easiest way to do that?

I have been using tox and I see there's an option for setting package to "wheel":

[testenv]
description = run the tests with pytest
package = wheel
wheel_build_env = .pkg

But that does not say how the wheel is produced; I am unsure if it

a) creates wheel directly from source tree
b) creates wheel from sdist which is created from source tree in a way which is identical to python -m build
c) creates wheel from sdist which is created from source tree in a way which differs from python -m build

Even if the answer would be c), the wheel tested by tox would not be the same wheel that would be uploaded, so it is not testing the correct thing. Most likely I should somehow give the wheel as an argument to tox / test runner.

Question

I want to create a wheel from sdist which is created from the source tree, and I want to run unit tests against the wheel(s) with multiple python versions. This is a pure python project, so there will be only a single wheel per version of the package. What would be the idiomatic way to run the tests against the same wheel(s) which I would upload to PyPI? Can I use tox for that?

Solution

The Tox 4.12.2 documentation on External Package builder tells that it is possible to define an external package option (thanks for the comment @JürgenGmach). The external package option means that you set

[testenv]
...
package = external

In addition to this, one must create a section called [.pkg_external] (or <package_env>_external if you have edited your package_env which has an alias isolated_build_env). In this section, one should define at least the package_glob, which tells the tox where to install the wheel. If you also want to create the wheel, the you can do that in the commands option of the [.pkg_external].

Simple approach (multiple builds)

Example of a working configuration (tox 4.12.2):

[testenv:.pkg_external]
deps =
    build==1.1.1
commands =
    python -c 'import shutil; shutil.rmtree("{toxinidir}/dist", ignore_errors=True)'
    python -m build -o {toxinidir}/dist

package_glob = {toxinidir}{/}dist{/}wakepy-*-py3-none-any.whl

Pros: Pretty simply to implement
Cons: This approach has the downside that you will trigger the build (python -m build) for each of your environments which do not have skip_install=True. This has an open issue: tox #2729.

Building wheel only once

It is also possible to make tox 4.14.2 build the wheel only once using the tox hooks. As can be seen from the Order of tox execution (in Appendix), one hook which can be used for this is the tox_on_install for the ".pkg_external" (either "requires" or "deps"). I use it to place a dummy file (/dist/.TOX-ASKS-REBUILD) which means that a build should be done. If that .TOX-ASKS-REBUILD exists, when the build script is ran, the /dist folder with all of its contents is removed, and new /dist folder with a .tar.gz and a .whl file is created.

Pros:
- Faster to run tox, as sdist and wheel built only as many times as required.
- Will build also if using tox with single env, like tox -e py311 (if not skip_install=True)
Cons:
- More involved
- Will not work in parallel mode. For that, probably would require to have separate build command which would be ran each time before tox (unless the parallelization plugin supports a common pre-command).

Hopefully this solution will become unnecessary at some point (when #2729 gets resolved)

The hook

Located in a toxfile.py at project root.

from __future__ import annotations

import typing
from pathlib import Path
from typing import Any

from tox.plugin import impl

if typing.TYPE_CHECKING:
    from tox.tox_env.api import ToxEnv


dist_dir = Path(__file__).resolve().parent / "dist"
tox_asks_rebuild = dist_dir / ".TOX-ASKS-REBUILD"


@impl
def tox_on_install(tox_env: ToxEnv, arguments: Any, section: str, of_type: str):
    if (tox_env.name != ".pkg_external") or (of_type != "requires"):
        return

    # This signals to the build script that the package should be built.
    tox_asks_rebuild.parent.mkdir(parents=True, exist_ok=True)
    tox_asks_rebuild.touch()

The tox_build_mypkg.py

Located at /tests/tox_build_mypkg.py

import shutil
import subprocess
from pathlib import Path

dist_dir = Path(__file__).resolve().parent.parent / "dist"


def build():
    if not (dist_dir / ".TOX-ASKS-REBUILD").exists():
        print("Build already done. skipping.")
        return

    print(f"Building sdist and wheel into {dist_dir}")
    # Cleanup. Remove all older builds; the /dist folder and its contents.
    # Note that tox would crash if there were two files with .whl extension.
    # This also resets the TOX-ASKS-REBUILD so we build only once.
    shutil.rmtree(dist_dir, ignore_errors=True)

    out = subprocess.run(
        f"python -m build -o {dist_dir}", capture_output=True, shell=True
    )
    if out.stderr:
        raise RuntimeError(out.stderr.decode("utf-8"))
    print(out.stdout.decode("utf-8"))


if __name__ == "__main__":

    build()

The tox.ini

[testenv]
; The following makes the packaging use the external builder defined in
; [testenv:.pkg_external] instead of using tox to create sdist/wheel.
; https://tox.wiki/en/latest/config.html#external-package-builder
package = external


[testenv:.pkg_external]
; This is a special environment which is used to build the sdist and wheel
; to the dist/ folder automatically *before* any other environments are ran.
; All of this require the "package = external" setting.
deps =
    ; The build package from PyPA. See: https://build.pypa.io/en/stable/
    build==1.1.1
commands =
    python tests/tox_build_mypkg.py

; This determines which files tox may use to install mypkg in the test
; environments. The .whl is created with the tox_build_mypkg.py
package_glob = {toxinidir}{/}dist{/}mypkg-*-py3-none-any.whl

Notes

This solution requires a fairly new version of tox. The tox_on_install hook was added in tox 4.0.9.
If using any tox extensions and having problems with the hooks being called, try without the tox extension first.
It is possible to also put the hooks in any installed python module and define the location in pyproject.toml as described in Extensions points. However, the toxfile.py is a bit handier as it does not have to be installed in the current environment.

Appendix

Order of tox execution

The order of execution within tox can be reverse-engineered by using the dummy hook file defined in the Appendix (tox_print_hooks.py) and the bullet point list about the order of execution in the System Overview. Note that I have set the package = external already which has some effect on the output. Here is what tox does:

1) CONFIGURATION
tox_register_tox_env
tox_add_core_config
tox_add_env_config (N+2 times[1])

2) ENVIRONMENT (for each environment)

tox_on_install (envname, deps)
envname: install_deps (if not cached)
    
If not all(skip_install) AND first time: [2]
  tox_on_install (.pkg_external, requires)
  .pkg_external: install_requires (if not cached)
  tox_on_install (.pkg_external, deps)
  .pkg_external: install_deps (if not cached)
      
If not skip_install:
  .pkg_external: commands  
  tox_on_install (envname, package) 
  envname: install_package [3]
    
tox_before_run_commands (envname)
envname: commands
tox_after_run_commands (envname)
  
tox_env_teardown (envname)

^[1] N = number of environments in tox config file. The "2" comes from .pkg_external and .pkg_external_sdist_meta
^[2] "First time" means: First time in this tox call. This is done only if there is at least one selected environment which does not have skip_install=True.
^[3] This installs the package from wheel. If using the package = external in [testenv], it takes the wheel from the place defined by the package_glob in the [testenv:.pkg_external]

The dummy hook file `tox_print_hooks.py`

from typing import Any

from tox.config.sets import ConfigSet, EnvConfigSet
from tox.execute.api import Outcome
from tox.plugin import impl
from tox.session.state import State
from tox.tox_env.api import ToxEnv
from tox.tox_env.register import ToxEnvRegister


@impl
def tox_register_tox_env(register: ToxEnvRegister) -> None:
    print("tox_register_tox_env", register)


@impl
def tox_add_core_config(core_conf: ConfigSet, state: State) -> None:
    print("tox_add_core_config", core_conf, state)


@impl
def tox_add_env_config(env_conf: EnvConfigSet, state: State) -> None:
    print("tox_add_env_config", env_conf, state)


@impl
def tox_on_install(tox_env: ToxEnv, arguments: Any, section: str, of_type: str):
    print("tox_on_install", tox_env, arguments, section, of_type)


@impl
def tox_before_run_commands(tox_env: ToxEnv):
    print("tox_before_run_commands", tox_env)


@impl
def tox_after_run_commands(tox_env: ToxEnv, exit_code: int, outcomes: list[Outcome]):
    print("tox_after_run_commands", tox_env, exit_code, outcomes)


@impl
def tox_env_teardown(tox_env: ToxEnv):
    print("tox_env_teardown", tox_env)