How to reference a requirements.txt in the pyproject.toml of a setuptools project?

I'm trying to migrate a setuptools-based project from the legacy setup.py towards modern pyproject.toml configuration.

At the same time I want to keep well established workflows based on pip-compile, i.e., a requirements.in that gets compiled to a requirements.txt (for end-user / non-library projects of course). This has important benefits as a result of the full transparency:

100% reproducible installs due to pinning the full transitive closure of dependencies.
better understanding of dependency conflicts in the transitive closure of dependencies.

For this reason I don't want to maintain the dependencies directly inside the pyproject.toml via a dependencies = [] list, but rather externally in the pip-compiled managed requirements.txt.

This makes me wonder: Is there a way to reference a requirements.txt file in the pyproject.toml configuration, without having to fallback to a setup.py script?

Solution

In setuptools 62.6 the file directive was made available for dependencies and optional-dependencies.

Use dynamic metadata:

[project]
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}

Note that the referenced file will use a requirements.txt-like syntax; each line must conform to PEP 508, so flags like -r, -c, and -e are not supported inside this requirements.txt. Also note that this capability is still technically in beta.

Additionally:

If you are using an old version of setuptools, you might need to ensure that all files referenced by the file directive are included in the sdist (you can do that via MANIFEST.in or using plugins such as setuptools-scm, please have a look on [sic] Controlling files in the distribution for more information).

Changed in version 66.1.0: Newer versions of setuptools will automatically add these files to the sdist.

If you want to use optional-dependencies, say, with a requirements-dev.txt, you will need to put an extra group, as follows (credit to Billbottom):

[project]
dynamic = ["dependencies", "optional-dependencies"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}
optional-dependencies = {dev = { file = ["requirements-dev.txt"] }}

However:

Currently, when specifying optional-dependencies dynamically, all of the groups must be specified dynamically; one can not specify some of them statically and some of them dynamically.