pythonpython-3.xpipartifactorypython-poetry

Is there a way to mimic pip --extra-index-url functionality for transitive dependency resolution?

In my corporate environment we are forced to proxy PyPI through our internal artifactory service, e.g.:

https://[COMPANY URL]/artifactory/api/pypi/pypi-release/simple

I therefore set up my pyproject.toml with this as the primary source:

[[tool.poetry.source]]
name = "artifactory_release"
url = "https://[COMPANY URL]/artifactory/api/pypi/pypi-release/simple"
priority = "primary"

The problem is that my company actually hosts multiple levels of artifactory for different types of internal builds. Most projects would get built & hosted in a different entirely firewalled internal repository, which does not have access to the wider PyPI:

https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/simple

Normally I would list this as a secondary/supplemental source, like so:

[[tool.poetry.source]]
name = "artifactory_unstable"
url = "https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/simple"
priority = "supplemental"

I would then expect to be able to pull packages specifically from this secondary repository by specifying the dependencies in my pyproject.toml file like so:

[tool.poetry.dependencies]
python = "^3.10 <3.12"
numpy = "1.23.5"
pandas = "1.4.3"
[MY PACKAGE] = {version = "[VERSION NO]", source = "artifactory_unstable"}

My problem is that when I try to pull some internally built packages from this secondary source, it looks like Poetry tries to resolve their dependencies (i.e. transitive dependencies) through that same secondary source, which does not have access to the wider PyPI. This therefore results in an error:

(base) my-computer: curr_dir$ poetry lock
Updating dependencies
Resolving dependencies... (10.6s)

  ValueError

  Package('[MY PACKAGE]', '[VERSION NO]') is not in list

  at ~/.local/pipx/venvs/poetry/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:66 in package
       62│         Note that this will be cached so the subsequent operations
       63│         should be much faster.
       64│         """
       65│         try:
    →  66│             index = self._packages.index(Package(name, version))
       67│ 
       68│             return self._packages[index]
       69│         except ValueError:
       70│             package = super().package(name, version, extras)

The following error occurred when trying to handle this error:

  AssertionError

  at ~/.local/pipx/venvs/poetry/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:526 in _fetch_content_length
      522│             # If we *could* download some file contents, then write them to the end of
      523│             # the file and set up our bisect boundaries by hand.
      524│             with self._stay():
      525│                 response_length = int(tail.headers["Content-Length"])
    → 526│                 assert response_length == min(initial_chunk_size, ret_length)
      527│                 self.seek(-response_length, io.SEEK_END)
      528│                 # Default initial chunk size is currently 1MB, but streaming content
      529│                 # here allows it to be set arbitrarily large.
      530│                 for chunk in tail.iter_content(CONTENT_CHUNK_SIZE):

When I use normal pip rather than Poetry, I can force the correct behaviour by specifying the primary PyPI index URL as an extra index URL for this specific package. This allows pip to pull my package from the internal firewalled repository, but to resolve its dependencies through the public PyPI proxy:

(base) my-computer: curr_dir$ python -m pip install --index-url https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/simple/ --extra-index-url https://[COMPANY URL]/artifactory/api/pypi/pypi-release/simple [MY PACKAGE]
Looking in indexes: https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/simple/, https://[COMPANY URL]/artifactory/api/pypi/pypi-release/simple
Collecting [MY PACKAGE]
  Using cached https://[COMPANY URL]/artifactory/api/pypi/python-internal-unstable/[DIR PATH]/[MY PACKAGE]-[VERSION NO]-py3-none-any.whl (26 kB)
Requirement already satisfied: azure-monitor-opentelemetry<2.0.0,>=1.6.5 in /Users/[ME]/Library/Caches/pypoetry/virtualenvs/[POETRY ENV]/lib/python3.11/site-packages (from [MY PACKAGE]) (1.6.10)
Collecting opencensus-ext-azure<2.0.0,>=1.1.4 (from [MY PACKAGE])
...
...
...
Downloading https://COMPANY URL/artifactory/api/pypi/pypi-release/packages/packages/.../.../pyasn1-0.6.1-py3-none-any.whl (83 kB)
Installing collected packages: opencensus-context, typing-inspection, python-dotenv, pydantic-core, ..., opencensus-ext-azure, [MY PACKAGE]
Successfully installed annotated-types-0.7.0 cachetools-5.5.2 ... opencensus-ext-logging-0.1.1 [MY PACKAGE]-[VERSION NO] ... rsa-4.9.1 typing-inspection-0.4.1

Is this possible to do with Poetry?

Solution

  • So it turns out this is a bug with Poetry 1.8.0 in relation to Artifactory that was patched in version 1.8.2.

    I upgraded my poetry version and now the pyproject.toml configuration I described above works as expected with no issues.