python-3.xvirtualenvsetuptoolssetup.pypython-wheel

How to tell pip3 to prefer download (and install) wheel files over the tar.gz when downloading and installing python3 packages


I'm a bit new to Python and all its "deployment" related tools: pip3,setuptools, virtualenv wheel etc, so I hope my question will make sense...anyways it is like so:

I have a Python3 project which is "managed" with virtual environment using virtualenv where all the projects' dependencies are "listed" within the project's setup.py file. The contents of the setup.py files are as follows:

setup(name="MyProjectName",
      version="0.1",
      description="Some description",
      url="someURL",
      author="My Name",
      author_email="someemail",
      license="MIT",
      packages=find_packages(),
      include_package_data=True,
      install_requires=["robotframework", "paramiko"])

As you can see, the only 3rd party packages the project uses (explicitly) are robotframework & paramiko.

Now when I'm deploying the project, I do the following actions (in that order):

  1. Create a virtual environment with the command: virtualenv -p python3 virtualEnvFolderName

  2. Switching "into" the virtual environment like so (I'm deploying it on a Linux machine): source virtualEnvFolderName/bin/activate

  3. Running the setup.py script with the install argument to "automatically" install all the project's dependencies with the following command: python3 setup.py install

--> Up until couple of days ago, all the 3rd party packages (and their "dependencies sub-packages") listed in the setup.py file where downloaded (and then installed) using their whl file, i.e. - for example:The output for the paramiko package installation would have been:

Reading https://pypi.org/simple/paramiko/ Downloading https://files.pythonhosted.org/packages/4b/80/74dace9e48b0ef923633dfb5e48798f58a168e4734bca8ecfaf839ba051a/paramiko-2.6.0-py2.py3-none-any.whl#sha256=99f0179bdc176281d21961a003ffdb2ec369daac1a1007241f53374e376576cf Best match: paramiko 2.6.0 Processing paramiko-2.6.0-py2.py3-none-any.whl Installing paramiko-2.6.0-py2.py3-none-any.whl to

--> This way, the installation was very quick (~1-3 seconds per package).

Today, when I performed the same procedure, on the same machine (I'm quiet sure I did not change any settings on my Ubuntu 16.04 machine), for each package the setup.py tried to install, it installed "via" the tar.gz file (i.e. sources ?) and NOT using the whl file --> which takes MUCH longer since for some of the packages it actually builds (complies) all the "underlying C libraries". This "change" makes my "installation procedure" execution time increase from ~20 seconds to ~4 minutes.

My questions are:

a) How can I resolve this situation - preferably without changing the deployment procedure, i.e. - still perform the 3 steps mentioned above, taking into account that perhaps one or more of the commands will be slightly modified (the creation of the virtual environment and/or some additional argument in required to the setup.py ? ).

b) If I have no other option, then using a pip3 install -r requirement.txt ... "procedure" will also be good, if it also will use whl file(s) whenever applicable.

c) If I will need to "switch" my virtual environment "generator" to venv it is OK (and actually preferred, in case it will deploy the project in the "same" duration).

NOTES: I tested it both on Ubuntu 16.04 and Ubuntu 18.04 machines with Python 3.5 and Python 3.6 respectively.

Thanks !!


Solution

  • You can now add the --prefer-binary argument to install from wheel instead of tar.gz (source). This prefers binary packages over source packages, even if the source packages are newer.

    Like this:

    pip install jellyfish --prefer-binary
    

    From the pip docs.