How to correct a Google Colab Pip installation error?

Using google colab, I am installing a few libraries used in an online example as shown in the following:

!pip install -q auto-gptq

This result in the following error:

ERROR: pip's dependency resolver does not currently take into account all the 
    packages that are installed. This behaviour is the source of the following 
    dependency conflicts.
cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 17.0.0 which is incompatible.
gcsfs 2024.6.1 requires fsspec==2024.6.1, but you have fsspec 2024.5.0 which is incompatible.
google-colab 1.0.0 requires requests==2.31.0, but you have requests 2.32.3 which is incompatible.
ibis-framework 8.0.0 requires pyarrow<16,>=2, but you have pyarrow 17.0.0 which is incompatible.

I tried the following to correct this depencdency error (it said 15.0.0a0 doesn't exist):

!pip install pyarrow==15.0.0

Which results in this new error:

ERROR: pip's dependency resolver does not currently take into account all the 
    packages  that are installed. This behaviour is the source of the following 
    dependency conflicts.
cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 15.0.0 which is incompatible.
Successfully installed pyarrow-15.0.0

Am I approaching correcting the original error the correct way? What should a do to get these error's resolved?

I did try to find some similar examples of the error on online but didn't find them helpful.

Solution

You can avoid the PyArrow version conflict between auto-gptq and Colab's default package versions by updating the cuDF version in Colab (see RAPIDS cuDF install instructions: https://docs.rapids.ai/install). As of this writing, Colab uses cuDF 24.04 with PyArrow 14. Installing cuDF 24.06 will get PyArrow 16.1, which avoids the PyArrow version conflicts with auto-gptq's datasets dependency.

Try the following:

!pip install --extra-index-url=https://pypi.nvidia.com cudf-cu12==24.6.*

!pip install auto-gptq

Then, you can see the installed versions:

import pyarrow; print(f"{pyarrow.__version__=}")
import cudf; print(f"{cudf.__version__=}")
import auto_gptq; print(f"{auto_gptq.__version__=}")

Output:

pyarrow.__version__='16.1.0'
cudf.__version__='24.06.01'
auto_gptq.__version__='0.7.1'

Here is a Colab notebook demonstrating the solver: https://colab.research.google.com/gist/bdice/90f95dc1f5bdecdd92f826c04f70aa03/auto-gptq-install.ipynb