I am developing a plugin-based application framework in C++ targeting linux/gcc. In my main application I use the classic approach of dlopen(), dlsym() functions to obtain an entry point into the plugin code.
This works well in the general case and for most plugins, including ones which are linked (dynamically when they are built) to other libraries.
One such plugin (call it libplugin.so) however needs to link to another library (libother.so). When I attempt to load it, the
dlopen("/path/to/libplugin.so", RTLD_LOCAL | RTLD_LAZY)
call fails, and dlerror() reports "cannot allocate memory in static TLS block".
Other questions here (e.g. this or this) and elsewhere led me to an effective workaround, which is to call my main executable with
LD_PRELOAD=/path/to/libother.so
Most of the results show this fix, but I could not find an explanation of the problem, nor why this fix works.
I would like to better understand the cause (or perhaps a list of possible causes) for why this particular library needs this workaround and others we have used do not. Could my dlopen flags be the culprit? Different combinations broke other stuff in the past.
I know that this library does indeed use Thread Local Storage, but I believe it's not the only one we use that does (though I'm not 100% sure on this). The very same libother.so works without LD_PRELOAD if instead of being linked in a plugin it is linked normally in a different application. I have access to the library source and can rebuild it, but it is quite complex and its support is limited.
P.S. Any workaround that can be "self-contained" within the build of libplugin.so, and not pollute env for my main, would also count as a solution for me.
I would like to better understand the cause (or perhaps a list of possible causes) for why this particular library needs this workaround and others we have used do not.
It is likely that libother.so (or some other shared library libother.so depends on) was linked with -ftls-model=initial-exec. When a shared library is linked that way, its access to thread-local storage is more efficient and faster, but at the cost of requiring that either the main executable is directly linked with that library, or reserves sufficient space in its initial TLS block for the library to use.
When you use LD_PRELOAD, you are effectively emulating the "main executable is linked directly with libother.so" -- the runtime loader will reserve enough space because it knows about libother.so from the process start.
You can read about tls here, and about various tls models here.