c++cposixcitrixdlsym

dlsym() + RTLD_NEXT doesn't work as expected on Ubuntu 20.04

I'm faced a strange runtime behavior on Ubuntu 20.04 (gcc v 9.3.0) when using dlsym() call.

Please, see below a simple example:

#include <iostream>
#include <dlfcn.h>
#include <execinfo.h>
#include <typeinfo>
#include <string>
#include <memory>
#include <cxxabi.h>
#include <cstdlib>

extern "C"
{
    void __cxa_throw(void *ex, void *info, void (*dest)(void *))
    {
        std::cout << "__cxa_throw() invoked \n";

        static void (*const rethrow)(void *, void *, void (*)(void *)) __attribute__((noreturn))
            = (void (*)(void *, void *, void (*)(void *)))dlsym(RTLD_NEXT, "__cxa_throw");

        std::cout << "addr in lib=" << &rethrow << "\n";

        rethrow(ex, info, dest);

        std::terminate();
    }
}


#include <iostream>

void foo()
{
  throw std::runtime_error("error");
}

int main()
{
    foo();
    return 0;
}

Build these 2 files as following:

g++ -fPIC -std=c++17 test.cpp -g -c -o test.o
g++ -shared ./test.o -o libtest.so
g++ main.cpp -std=c++17 -g -pedantic -L./ -ltest -ldl

Feeding ldd to ./a.out gives :

ldd a.out 
    linux-vdso.so.1 (0x00007ffe01186000)
    /usr/local/lib/AppProtection/libAppProtection.so (0x00007f1dbd738000)
    libtest.so (0x00007f1dbd733000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dbd708000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1dbd526000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1dbd50b000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dbd319000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dbd2f4000)
    libX11.so.6 => /lib/x86_64-linux-gnu/libX11.so.6 (0x00007f1dbd1b7000)
    libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f1dbd18d000)
    libXi.so.6 => /lib/x86_64-linux-gnu/libXi.so.6 (0x00007f1dbd17b000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1dbd964000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1dbd02c000)
    libXau.so.6 => /lib/x86_64-linux-gnu/libXau.so.6 (0x00007f1dbd024000)
    libXdmcp.so.6 => /lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f1dbd01c000)
    libXext.so.6 => /lib/x86_64-linux-gnu/libXext.so.6 (0x00007f1dbd007000)
    libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f1dbcfed000)

We can see that libtest.so is resolved before libstdc++.so. My expectation how this code should work :

This works as expected on all platforms except Ubuntu 20.04 where rethrow is referencing the __cxa_throw from libtest.so (but not libstc++.so) and thus causing endless recursion.

Please assist as I'm puzzled with runtime behavior.

Solution

  • The "app protection" component of the Citrix ICA client installs the library /usr/local/lib/AppProtection/libAppProtection.so and adds an entry for it to /etc/ld.so.preload, causing it to be loaded into every dynamically-linked process. Among other things, this library replaces the dlsym function with its own. (If you're curious how this doesn't just always break everything by going into an infinite loop, see How can I intercept dlsym calls using LD_PRELOAD?. It actually seems like Citrix's code may have been copied and pasted straight from that answer.) The problem is that since RTLD_NEXT depends on being able to inspect the return address, special care is required to avoid breaking that when hooking dlsym, and they didn't take that special care. As a result, RTLD_NEXT will look for the symbol in the next library after libAppProtection.so, instead of in the next library after your code, which causes exactly the problem you ran into.

    Here's some choices for what to do about it: