pythonpipconflicting-libraries

How to resolve python libraries dependencies when using pip

I have come across some nasty dependencies.

Looking around I found solutions like upgrade this, downgrade that...

Some solutions work for some but not for others.

Is there a more 'rounded' way to tackle this issue?

Solution

  • First you need to understand that pip can resolve problems one at a time and when you put it in a corner, it can't go further.

    But, if you give to pip the 'big problem' it has a nice way to try to resolve it. It may not always work, but for most cases it will.

    The solutions you normally find out there are in some cases a coincidence. Someone has an environment similar to another person and one solution works for both.

    But if the solution takes into consideration 'your present environment' then the solution should work for more people than just 'the coincidences'.

    Disclaimer: below are linux/terminal commands.

    1. Upgrade pip. We want the smartest pip we can get.

    pip install --upgrade pip

    1. Extract the list of packages you want to install.

    In my case (these and many others, trimmed for brevity)

    google-cloud-texttospeech attrdict google-cloud-language transformers

    1. Give them all at once to pip.

    pip install google-cloud-texttospeech attrdict google-cloud-language transformers

    It will try all the combinations of versions and dependencies' versions until it finds something suitable. This will potentially download a ton of packages just to see their dependencies, so you only want to make this once.

    enter image description here

    1. If happy with the result, extract the requirements file.

    pip freeze > requirements.txt

    This contains all the packages installed, we are not interested in all.

    And from it, extract the specific versions of your desired packages.

    cat requirements.txt | egrep -i "google-cloud-texttospeech|attrdict|google-cloud-language|transformers"

    Note: The command above may not list all your required packages, just the ones that appear in the requirements.txt, so check that all show up.

    attrdict==2.0.1
google-cloud-language==1.2.0
google-cloud-texttospeech==2.12.3
transformers==2.11.0

    Now you can put that on a file like resolved-dependencies.txt

    And next time, install the packages directly with the valid & compatible version with.

    pip install -r resolved-dependencies.txt