In my corporate environment we are forced to proxy PyPI through our internal artifactory service, e.g.:
https://[COMPANY URL]/artifactory/api/pypi/pypi-release/simple
I therefore set up my pyproject.toml
with this as the primary source:
[[tool.poetry.source]]
name = "artifactory_release"
url = "https://[COMPANY URL]/artifactory/api/pypi/pypi-release/simple"
priority = "primary"
The problem is that my company actually hosts multiple levels of artifactory for different types of internal builds. Most projects would get built & hosted in a different entirely firewalled internal repository, which does not have access to the wider PyPI:
https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/simple
Normally I would list this as a secondary/supplemental source, like so:
[[tool.poetry.source]]
name = "artifactory_unstable"
url = "https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/simple"
priority = "supplemental"
I would then expect to be able to pull packages specifically from this secondary repository by specifying the dependencies in my pyproject.toml
file like so:
[tool.poetry.dependencies]
python = "^3.10 <3.12"
numpy = "1.23.5"
pandas = "1.4.3"
[MY PACKAGE] = {version = "[VERSION NO]", source = "artifactory_unstable"}
My problem is that when I try to pull some internally built packages from this secondary source, it looks like Poetry tries to resolve their dependencies (i.e. transitive dependencies) through that same secondary source, which does not have access to the wider PyPI. This therefore results in an error:
(base) my-computer: curr_dir$ poetry lock
Updating dependencies
Resolving dependencies... (10.6s)
ValueError
Package('[MY PACKAGE]', '[VERSION NO]') is not in list
at ~/.local/pipx/venvs/poetry/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:66 in package
62│ Note that this will be cached so the subsequent operations
63│ should be much faster.
64│ """
65│ try:
→ 66│ index = self._packages.index(Package(name, version))
67│
68│ return self._packages[index]
69│ except ValueError:
70│ package = super().package(name, version, extras)
The following error occurred when trying to handle this error:
AssertionError
at ~/.local/pipx/venvs/poetry/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:526 in _fetch_content_length
522│ # If we *could* download some file contents, then write them to the end of
523│ # the file and set up our bisect boundaries by hand.
524│ with self._stay():
525│ response_length = int(tail.headers["Content-Length"])
→ 526│ assert response_length == min(initial_chunk_size, ret_length)
527│ self.seek(-response_length, io.SEEK_END)
528│ # Default initial chunk size is currently 1MB, but streaming content
529│ # here allows it to be set arbitrarily large.
530│ for chunk in tail.iter_content(CONTENT_CHUNK_SIZE):
When I use normal pip rather than Poetry, I can force the correct behaviour by specifying the primary PyPI index URL as an extra index URL for this specific package. This allows pip to pull my package from the internal firewalled repository, but to resolve its dependencies through the public PyPI proxy:
(base) my-computer: curr_dir$ python -m pip install --index-url https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/simple/ --extra-index-url https://[COMPANY URL]/artifactory/api/pypi/pypi-release/simple [MY PACKAGE]
Looking in indexes: https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/simple/, https://[COMPANY URL]/artifactory/api/pypi/pypi-release/simple
Collecting [MY PACKAGE]
Using cached https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/[DIR PATH]/[MY PACKAGE]-[VERSION NO]-py3-none-any.whl (26 kB)
Requirement already satisfied: azure-monitor-opentelemetry<2.0.0,>=1.6.5 in /Users/[ME]/Library/Caches/pypoetry/virtualenvs/[POETRY ENV]/lib/python3.11/site-packages (from [MY PACKAGE]) (1.6.10)
Collecting opencensus-ext-azure<2.0.0,>=1.1.4 (from [MY PACKAGE])
...
...
...
Downloading https://COMPANY URL/artifactory/api/pypi/pypi-release/packages/packages/.../.../pyasn1-0.6.1-py3-none-any.whl (83 kB)
Installing collected packages: opencensus-context, typing-inspection, python-dotenv, pydantic-core, ..., opencensus-ext-azure, [MY PACKAGE]
Successfully installed annotated-types-0.7.0 cachetools-5.5.2 ... opencensus-ext-logging-0.1.1 [MY PACKAGE]-[VERSION NO] ... rsa-4.9.1 typing-inspection-0.4.1
Is this possible to do with Poetry?
So it turns out this is a bug with Poetry 1.8.0 in relation to Artifactory that was patched in version 1.8.2.
Bug details: https://github.com/python-poetry/poetry/issues/9056
Changelog details for 1.8.2: https://python-poetry.org/history/#182---2024-03-02
I upgraded my poetry version and now the pyproject.toml
configuration I described above works as expected with no issues.