pythontensorflowcudanvidiaarchlinux

Tensorflow errors: cuFFT, cuDNN, cuBLAS and "Assertion '__n < this->size()' failed"


I just started with TF and Keras and found out that I can't run these on my computer. I've notice the problem first in jupyter notebook, and than recreate it in a python file.

The code to reproduce (main.py):

import os
os.environ["KERAS_BACKEND"] = "tensorflow"
import keras
from keras import layers
from keras import ops


print("pass")    
model = keras.Sequential()
model.add(layers.Input(shape=(28,)))
print("pass")

The python main.py output:

2024-10-16 22:30:52.054073: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1729081852.071578   41750 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1729081852.076640   41750 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
pass
/usr/include/c++/14.1.1/bits/stl_vector.h:1130: constexpr std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type) [with _Tp = pybind11::object; _Alloc = std::allocator<pybind11::object>; reference = pybind11::object&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
fish: Job 1, 'python main.py' terminated by signal SIGABRT (Abort)

I using arch linux and installed python-tensorflow-opt-cuda package. I have a nvidia card and working cuda.

The nvcc --version output:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Sep_12_02:18:05_PDT_2024
Cuda compilation tools, release 12.6, V12.6.77
Build cuda_12.6.r12.6/compiler.34841621_0

The python -c "import tensorflow as tf; print(tf.__version__)" output:

2024-10-16 22:35:45.309836: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1729082145.328052   42393 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1729082145.333326   42393 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2.18.0-rc1

The pacman -Q cuda output:

cuda 12.6.2-1

The pacman -Q cudnn output:

cudnn 9.2.1.18-1

The pacman -Q nvidia output:

nvidia-dkms 560.35.03-14

nvidia-dkms 560.35.03-14 The pacman -Q blas output:

blas 3.12.0-5

The pacman -Q python-tensorflow-opt-cuda output:

python-tensorflow-opt-cuda 2.18rc1-2

The pacman -Q python output:

python 3.12.7-1

The pacman -Q python-numpy output:

python-numpy 2.1.2-1

The pacman -Q python-keras output:

python-keras 3.4.1-1

The sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1650        Off |   00000000:01:00.0 Off |                  N/A |
| N/A   43C    P8              1W /   50W |       8MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+
Kernel: 6.11.3-zen1-1-zen
CPU: AMD Ryzen 5 4600H with Radeon Graphics (12) @ 3.000GH
GPU: NVIDIA GeForce GTX 1650 Mobile / Max-Q

I also don't understand why tf is 2.18 version in archlinux package repo. Isn't it unstable for now?

I've tried to reinstall all of the packages, but didn't downgrade them yet. I also tried using docker version of tensorflow, and I still got the same errors.


Solution

  • Found the problem.

    It was an outdated python-optree package. Seems that 0.13.0 version doesn't have the issue.

    The issue was:

    >>> from optree import *
    >>> tree_map(lambda x: x, ())
    /usr/include/c++/14.1.1/bits/stl_vector.h:1130: constexpr std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type) [with _Tp = pybind11::object; _Alloc = std::allocator<pybind11::object>; reference = pybind11::object&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
    fish: Job 1, 'python' terminated by signal SIGABRT (Abort)
    

    Empty tuples or arrays raised the issue.

    In version 0.13.0:

    >>> from optree import *
    >>> tree_map(lambda x: x, ())
    ()
    

    It's not throwing anymore and everything works.