I have a Yocto Kirkstone Aarch64 SDK containing GCC 11.5, and I would like to build something with a GCC plugin using that SDK. I have compiled the plugin, and attempting to use it yields
error: symbol lookup error: undefined symbol: _ZTVN4json14integer_numberE (fatal)
This SDK is being produced along with a full embedded linux image, which I have the ability to modify. However, I would like to be able to build plugins and use them with the SDK without having to modify the layers of the build.
I have validated that this same plugin can be built, and used, with GCC 11.4 and 12.3 on my host system (Ubuntu 22.04.5)
The project in question is https://github.com/royjacobson/externis. This project does not natively support a "cross-compilation" sort of usage, but I've modified it locally such that I can override the GCC plugin include directory - building with something like
cmake -DGCC_PLUGIN_INCLUDE=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/lib/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/plugin
Note - this is done using my host system's compiler; I am not using the SDK tooling to build the plugin.
The plugin header is present, and it builds just fine. I can install the plugin to that directory, and pass -fplugin=externis to GCC while using the SDK's environment to activate the plugin; however, when cc1plus attempts to load it, symbols fail to resolve.
My initial thought was to validate that the symbols exist somewhere at all - this has been validated. I'm no expert on the GCC plugin infrastructure, but as far as I can tell, the symbols are built into cc1plus directly, and then resolve from there during a dlopen call for the plugin being loaded. See https://github.com/gcc-mirror/gcc/blob/master/gcc/plugin.cc#L705
Using readelf, I have observed that these symbols are present both in my host cc1plus and the cross-canadian one in the SDK
$ readelf -aW /lib/gcc/x86_64-linux-gnu/12/cc1plus | grep _ZTVN4json14integer_numberE
13196: 0000000002345d70 48 OBJECT WEAK DEFAULT 18 _ZTVN4json14integer_numberE
$ readelf -aW /opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/cc1plus | grep _ZTVN4json14integer_numberE
29612: 0000000001a97e18 48 OBJECT WEAK DEFAULT 17 _ZTVN4json14integer_numberE
I've also attempted to build something using LD_DEBUG=all - and in this case, I can observe that my host's cc1plus provides this symbol, but for some reason the one in the toolchain doesn't. Below are the relevant sections of the output from ld
218809: symbol=_ZTVN4json14integer_numberE; lookup in file=/usr/lib/gcc/x86_64-linux-gnu/12/cc1plus [0]
218809: binding file /usr/lib/gcc/x86_64-linux-gnu/12/plugin/externis.so [0] to /usr/lib/gcc/x86_64-linux-gnu/12/cc1plus [0]: normal symbol `_ZTVN4json14integer_numberE'
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/cc1plus [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/../../../../../lib/libmpc.so.3 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/../../../../../lib/libmpfr.so.6 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/../../../../../lib/libgmp.so.10 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/../../../../../lib/libz.so.1 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/../../../../../lib/libzstd.so.1 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/lib/libm.so.6 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/lib/libc.so.6 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/lib/ld-linux-x86-64.so.2 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/lib/libpthread.so.0 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/lib/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/plugin/externis.so [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/../../../../../lib/libstdc++.so.6 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/lib/libgcc_s.so.1 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/lib/libc.so.6 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/lib/libm.so.6 [0]
218392: symbol=_ZTVN4json14integer_numberE; lookup in file=/opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/lib/ld-linux-x86-64.so.2 [0]
218392: /opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/lib/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/plugin/externis.so: error: symbol lookup error: undefined symbol: _ZTVN4json14integer_numberE (fatal)
The most obvious hypothesis here is that I just don't understand something about symbol tables and the symbols are either not present, partially present, or present but in some way that they don't match - maybe symbol versions are coming into play? But I would have expected symbol versions to manifest in the readelf output
Another likely option is that something about how GCC is being built is shooting this in the foot - I'm using --enable-plugin, and I've explicitly disabled stripping of the package; but I'm not going to rule out that something in the Yocto build process is still messing with the binary in a way that causes the symbols to not be valid.
I've also been looking into if I could/should add extra tooling in the SDK that can be used to build the plugin itself targeting the SDK.
The next thing I'm planning to attempt is just adding the plugin directly to the SDK - add a recipe to my Yocto build, package it up, etc. But I'd like to avoid that if possible as a long-term solution, or at the very least, I'm looking to understand what it is about my strategy isn't wrong.
Problem solved!
My initial analysis was missing an important detail - the readelf command I was running didn't provide a complete picture; the symbols were present, yes, but not in the location that dlopen searches.
If we modify the search string, we see slightly different results when trying to find our symbols:
$ readelf -aW ./lib/gcc/x86_64-linux-gnu/11/cc1plus | grep -E "Symbol table|_ZTVN4json14integer_numberE"
Symbol table '.dynsym' contains 24102 entries:
10292: 00000000018a3590 48 OBJECT GLOBAL DEFAULT 17 _ZTVN4json14integer_numberE
$ readelf -aW /opt/toolchains/kirkstone/aarch64/sysroots/x86_64-sdk-linux/usr/libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.5.0/cc1plus | grep -E "Symbol table|_ZTVN4json14integer_numberE"
Symbol table '.symtab' contains 59834 entries:
29612: 0000000001a97e18 48 OBJECT WEAK DEFAULT 17 _ZTVN4json14integer_numberE
The working cc1plus has the symbol in .dynsym, and the broken one has it in .symtab
Ultimately, the solution was to incorporate this change in GCC 14 into the software in the SDK - https://github.com/gcc-mirror/gcc/commit/4d9bc81a5d8d884dee7a7781fa4c1577a6c9681a.
For anyone not familiar with Yocto, it falls into this situation exactly - you have a build machine that produces both an SDK for some other machine (the host) and for the target. The "cross-canadian" build of GCC is the one that goes into the SDK - built on machine A, targeting machine B, to cross-compile for machine C.