How do I diagnose why a shared library is not found?

I am trying to run an app which is linked against libboost_system.so

$ ./app
app: error while loading shared libraries: libboost_system.so.1.63.0: cannot open shared object file: \
     No such file or directory

If I run ldd on the app, it shows it can't find the library:

$ ldd ./app
    linux-vdso.so.1 (0x00007ffdedb94000)
    libboost_system.so.1.63.0 => not found
    libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f19a1a7b000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f19a1a70000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f19a1a69000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f19a1879000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f19a1733000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f19a1716000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f19a16f4000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f19a152a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f19a1ae5000)

The library is present on my system:

$ locate libboost_system.so.1.63.0
/usr/local/lib/libboost_system.so.1.63.0

$ ls -la /usr/local/lib/libboost_system*
-rw-r--r--. 1 root root 47014 Jul  2 16:57 /usr/local/lib/libboost_system.a
lrwxrwxrwx. 1 root root    25 Jul  2 16:57 /usr/local/lib/libboost_system.so -> libboost_system.so.1.63.0
-rwxr-xr-x. 1 root root 19816 Jul  2 16:57 /usr/local/lib/libboost_system.so.1.63.0

ld is configured with /usr/local/lib:

$ ld --verbose | grep SEARCH_DIR | sed 's/; /\n/g'
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib64")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib")
SEARCH_DIR("=/usr/local/lib")                    <----- here
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")

I've also run ldconfig to refresh the cache, but that doesn't help.

I tried searching for RPATH or RUNPATH with readelf, neither are specified:

$ readelf -d ./app | grep -i path
< no results >

If I explicitly set LD_LIBARY_PATH then the library is found:

$ LD_LIBRARY_PATH=/usr/local/lib ldd ./app
    linux-vdso.so.1 (0x00007fffa9a5d000)
    libboost_system.so.1.63.0 => /usr/local/lib/libboost_system.so.1.63.0 (0x00007f5fff664000)
    libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f5fff5fc000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f5fff5f1000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f5fff5ea000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f5fff3fa000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f5fff2b4000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5fff297000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5fff275000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f5fff0ab000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5fff66b000)

Why can't ld find libboost_system.so.1.63.0 without specifying LD_LIBRARY_PATH?

Solution

You are looking at the "wrong" linker. ld is the static linker and isn't what's used by at runtime to locate the shared libraries.

The dynamic linker, ld.so, is what locates shared libraries.

The dynamic linker does look at the LD_LIBRARY_PATH and that's why it works if you set it.

If you want to update the search locations for dynamic linker (and not want to set LD_LIBRARY_PATH), you can create a file in /etc/ld.so.conf.d/ and update /etc/ld.so.cache with ldconfig.