python-3.xrapidscudf

cudf instllation issue on centos7

I'm new to rapids ai libraries. I've an existing conda environment yaml file where I'm using python 3.8.5, tensorflow 2.7.0, opencv-python-headless 4.5.5.62, numpy 1.22.2, pandas 1.4.1, pandas-profiling 3.1.0, seaborn 0.11.2, matplotlib 3.5.1, jupyterlab 3.2.9.

I've added below 2 channels to the file:

And the below packages:

The installation is going on for hours and while trying to find incompatible packages, it seems to be in some sort of loop as I keep seeing below message multiple times in the terminal:

Found conflicts! Looking for incompatible packages.

Is there any known issue/limitations that I should be aware of?

Since we don't get interactive shell on GPU h/w easily, I'm trying the conda environment update on non-GPU machine and once installed, I'll try cudf package on GPU machine.

EDIT1: This is what I have as working without tensorflow and tensorflow-hub

    name: cudf-env
        channels:
          - default
          - rmg
          - rapidsai
          - nvidia
          - numba
          - conda-forge
          - anaconda
        dependencies:
          - glibc=2.19
          - libgcc-ng=11.2.0
          - python=3.8.5
          - cudf=22.02
          - cudatoolkit=11.2
          - pytest=6.1.2
          - pandas=1.3.5
          - numpy=1.21.5
          - requests=2.25.0
          - scikit-learn=0.24.2
          - dill=0.3.4
          - tqdm=4.62.3
          - ruamel.yaml=0.17.19
          - yappi=1.3.3
          - black=22.1.0
          - pillow=9.0.1
          - jupyterlab=3.2.9
          - matplotlib=3.5.1
          - seaborn=0.11.2
          - plotly=5.6.0
          - pandas-profiling=3.1.0
          - black=22.1.0
        #  - pip
        #  - pip:
        #      - tensorflow==2.7.0
        #      - tensorflow-hub==0.12.0
        #      - opencv-python-headless==4.5.5.62
        #      - opencv-contrib-python-headless==4.5.5.62

Now, if I uncomment the pip section, the anaconda crashes while creating the environment. Since pip may not be supported with cudf, I tried following as well, the conda create env hangs while solving the environment (strangely, it's not resolving from conda-forge channel):

    name: cudf-env
        channels:
          - default
          - rmg
          - rapidsai
          - nvidia
          - numba
          - conda-forge
          - anaconda
        dependencies:
          - glibc=2.19
          - libgcc-ng=11.2.0
          - python=3.8.5
          - cudf=22.02
          - cudatoolkit=11.2
          - pytest=6.1.2
          - pandas=1.3.5
          - numpy=1.21.5
          - requests=2.25.0
          - scikit-learn=0.24.2
          - dill=0.3.4
          - tqdm=4.62.3
          - ruamel.yaml=0.17.19
          - yappi=1.3.3
          - black=22.1.0
          - pillow=9.0.1
          - jupyterlab=3.2.9
          - matplotlib=3.5.1
          - seaborn=0.11.2
          - plotly=5.6.0
          - pandas-profiling=3.1.0
          - black=22.1.0
          - tensorflow
          - tensorflow-hub

My system details are following:

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"

$ uname -r
3.10.0-1127.10.1.el7.x86_64

EDIT2: I forgot to mention that if I comment out glibc, cudf and cudatoolkit, the tensorflow installation through pip works fine.

Solution

  • Challenges updating conda environments can be tricky to untangle, but in this case the issue (in terms of cuDF) is likely the pinning of pandas to 1.4.1.

    cuDF does not yet support pandas=1.4.1. The cuDF nightly packages currently support pandas >=1.0,<1.4.0dev0 (everything from 1.0 up to a dev build of 1.4.0). For the moment, if you switch to pandas=1.3.5 things will work (assuming the other packages are compatible -- Tensorflow likely will require CUDA Toolkit 11.2, not 11.5, and some others might not be either). You can also let conda solve for the appropriate pandas version.