pythonpandasazureazure-devopsplotly-dash

Python (Dash) web app deployment on Azure with pipeline running too long with message Building wheel for pandas still running. How to optimize?


I have a Python web app (Dash) with the following requirements.txt:

dash==2.17.0
numpy==1.26.4
pandas==1.5.3
dash-mantine-components==0.14.3
python-dotenv==1.0.1
dash-iconify==0.1.2
dash-bootstrap-components==1.6.0
pyspark==3.5.1
pytz==2024.1
package @ git+https://${GITHUB_TOKEN}:x-oauth-basic@github.com/package
...

and within this package, this is the pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pandas = "==1.5.3"
pyspark = "==3.5.1"
... other dependencies

[dev-packages]

[requires]
python_version = "3.9"

And on Azure App Service, this is the configuration: Python stack with Python version 3.12.

But when I run my pipeline on Azure DevOps to build and deploy my web app, it is taking at least 20 minutes for each stage. Primarily with this step taking so long (at least around 10-12 mins):

  Building wheel for pandas (pyproject.toml): started
  Building wheel for pandas (pyproject.toml): still running...
  Building wheel for pandas (pyproject.toml): still running...
  Building wheel for pandas (pyproject.toml): still running...
  Building wheel for pandas (pyproject.toml): still running...
  Building wheel for pandas (pyproject.toml): still running...
  Building wheel for pandas (pyproject.toml): still running...
  Building wheel for pandas (pyproject.toml): still running...
  Building wheel for pandas (pyproject.toml): still running...
  Building wheel for pandas (pyproject.toml): still running...
  Building wheel for pandas (pyproject.toml): still running...

Lastly, this is the code I used to install python dependencies in my build&deploy.yml file:

- script: |
        python -m venv antenv
        source antenv/bin/activate
        python -m pip install --upgrade pip
        pip install setup
        pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
      displayName: 'Install Python dependencies'

Furthermore, the build&deploy uses python version 3.12, versionSpec: '3.12'

Is there a way to optimize this Python app deployment on Azure with Azure DevOps pipeline? Am I doing something wrong in terms of versioning and compatibility?


Solution

  • I can reproduce the same issue when using pandas==1.5.3 and Python 3.12.

    pandas==1.5.3 does not wheels built for your version, so your system builds them for itself each time. It will take long time to complete. You can check the available download files from here.

    To solve this issue, you can upgrade the pandas version to 2.2.2.

    For example:

    dash==2.17.0
    numpy==1.26.4
    pandas==2.2.2
    dash-mantine-components==0.14.3
    python-dotenv==1.0.1
    dash-iconify==0.1.2
    dash-bootstrap-components==1.6.0
    pyspark==3.5.1
    pytz==2024.1
    

    Then the package install step will be optimized.

    Result:

    enter image description here

    For more detailed info, you can refer to this release note: pandas 2.2.2

    Update:

    Since you are required to use pandas <2.2.0, you can use the version Pandas version 2.1.4.

    For example:

    pandas==2.1.4
    

    For more detailed info, you can refer to the download files in pandas 2.1.4

    Result:

    enter image description here