[SOLVED] Fail to load library in KDB kernel jupyter notebook

Fail to load library in KDB kernel jupyter notebook

I have installed the arrowkdb library, and use it to load parquet files in KDB. My $QHOME directory now contains:

├── kc.lic
├── arrowkdb.q 
├── l64
│   ├──arrowkdb.so
│   └── q
└── q.k

But when use \l to load arrowkdb.q, it raised following errors:

evaluation error:

libarrow.so.900: cannot open shared object file: No such file or directory

  [3]  /home/jiahe/q/arrowkdb.q:5: dt.na:`arrowkdb 2:(`null;1);
                                                   ^

  [0]  \l arrowkdb.q
       ^

So far checked:

The module works well when start from console:

~/q$ q arrowkdb.q 
KDB+ 4.1 2024.07.08 Copyright (C) 1993-2024 Kx Systems
l64/ 16(24)core 579514MB xxxx ubuntu-xxxxx-amd64-base xx.xx.xx.xx EXPIRE 2025.06.15 xxxxxxx@gmail.com KDB PLUS TRIAL #5020784

q)show t: .arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
1i
q)

I can load other q-scripts in $QHOME with \l in the notebook: notebook interface

I am new to KDB and also know little about Apache Arrow. Thank you very much for any suggestions:)

EDIT: I have tested several more times and find following patterns:

In my kdb conda environment, where I installed required packages to run KDB notebook, \l arrowkdb.q will raise the same error in q console as in the notebook.
While in base environment, \l arrowkdb.q works normally. Is there anyway to test whether the problem is caused by anaconda environment?

Solution

It is a dynamic load function "2:" https://code.kx.com/q/ref/dynamic-load/.

So it will look for libraries in $LD_LIBRARY_PATH if you use Linux. Please note the error you posted says that you don't have libarrow.so library, not arrowkdb.so You need to put all missed *.so libraries (correctly built for your platform) to $LD_LIBRARY_PATH and it will work.

arrowkdb "Third-party library installation" is here https://github.com/KxSystems/arrowkdb/tree/main?tab=readme-ov-file#third-party-library-installation