pythonazureazure-devopspypitwine

Publishing package to PyPi from Azure Devops hangs at twine upload

I want to publish a built .whl package to PyPi.org from our Azure Devops release pipeline, but the script twine upload keeps hanging without error, completion or time out. So the actual upload of our package (which is extremely small) does not work.

This is how it is set up:

The yaml build:

trigger:
- master
pr: none
pool:
  vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.7'
    addToPath: true
    architecture: 'x64'

- script: python -m pip install --upgrade pip setuptools wheel
  displayName: 'Install tools'

- script: pip install -r src/requirements.txt
  displayName: 'Install requirements'

- script: |    
    python src/setup.py bdist_wheel 
  displayName: 'Artifact creation'

- script: python -m pip install --upgrade twine
  displayName: 'Install Twine'

- task: TwineAuthenticate@1
  inputs:
    pythonUploadServiceConnection: 'AzureML PyPi feed'

- script: |
   python -m twine upload --config-file $(PYPIRC_PATH) dist/*.whl
  displayName: 'Publish to PyPi through Twine'

The service connection:

The release logs (most relevant parts):

TwineAuthenticate step:

Starting: TwineAuthenticate
==============================================================================
Task         : Python twine upload authenticate
Description  : Authenticate for uploading Python distributions using twine. Add '-r FeedName/EndpointName --config-file $(PYPIRC_PATH)' to your twine upload command. For feeds present in this organization, use the feed name as the repository (-r). Otherwise, use the endpoint name defined in the service connection.
Version      : 1.165.0
Author       : Microsoft Corporation
Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/package/twine-authenticate
==============================================================================
75c64e89-xxxxxredactedxxxxx exists true
Adding authentication to configuration for registry azure-pypi
Successfully added auth for 0 internal feed and 1 external endpoint.
Finishing: TwineAuthenticate

Publishing step:

Starting: Publish to PyPi through Twine
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.164.0
Author       : Microsoft Corporation
Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
python -m twine upload --config-file /home/vsts/work/_temp/twineAuthenticate/Wv8nMR/.pypirc dist/*.whl
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/0d772e31-148f-4e3c-999b-b2a43a02b287.sh
Uploading distributions to https://upload.pypi.org/legacy/

This keeps unchanged for more than 30 minutes, so I just cancel my release without the package being deployed. Any idea on this?

Solution

  • I have a temporary workaround for now:

    I skipped the usage of a config file and executed the following code instead:

    - script: |
   python -m twine upload --skip-existing --verbose -p $(pypi-api-token) -u __token__ --repository $(pypi-project-name) --repository-url https://upload.pypi.org/legacy/ dist/*.whl
  displayName: 'Publish to PyPi through Twine'

    And there, I noticed a bit more (and better) exception logging, that pointed me out two things:

    1. The token I was using for my config file was set to a project scope with a different name than the project I specified in my Service Connection
    2. The repository url really has to end with the trailing forward slash, because otherwise you get a RedirectDetected exception.

    Based on the above findings, I updated my yaml snippet to the following:

    - script: |
   python -m twine upload --skip-existing --verbose --repository $(pypi-project-name) --config-file $(PYPIRC_PATH) dist/*.whl
  displayName: 'Publish to PyPi through Twine'

    But when executing this, I now get the following exception message in the build:

    Generating script.
Script contents:
python -m twine upload --skip-existing --verbose --repository arcus-azureml --config-file /home/vsts/work/_temp/twineAuthenticate/2QGKVH/.pypirc dist/*.whl
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/7605dac9-5fa9-4856-94af-e938018278a5.sh
Uploading distributions to https://upload.pypi.org/legacy/
Uploading arcus_azureml-0.0.2-py3-none-any.whl

  0%|          | 0.00/5.80k [00:00<?, ?B/s]
100%|██████████| 5.80k/5.80k [00:00<00:00, 52.4kB/s]HTTPError: 403 Client Error: Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details for url: https://upload.pypi.org/legacy/

Content received from server:
<html>
 <head>
  <title>403 Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details</title>
 </head>
 <body>
  <h1>403 Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details</h1>
  Access was denied to this resource.<br/><br/>
Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details

    I also outputted the contents of the pypirc file, using the CAT command and the contents of the file are like this:

    [distutils]
index-servers=arcus-azureml 
[arcus-azureml]
repository=https://upload.pypi.org/legacy/
username=build
password=***

    Updated solution

    So, the fix to have the 403 arranged was to change my Service Connection to use 'Username and Password' as authentication method , instead of Authentication Token, with the following settings:

    After performing this, everything was working the way I wanted it.