I am trying to use the function EVP_PKEY_new_raw_private_key from libcrypto.so.3.
When I link with -l:libcrypot.so.3, it works.
When I try to open the same file with dlmopen+dlsym, it SEGV when the function is called.
An MWE is here:
main.c
#define _GNU_SOURCE
#include <assert.h>
#include <dlfcn.h>
#include <openssl/evp.h>
// c.f. objects/objects.pl
#define NID_X25519 1034
// some data to call new with
unsigned const char scalar[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x10, 0x11,
0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x20, 0x21, 0x22,
0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x30, 0x31, 0x32,
};
int main(int argc, char **argv) {
int keylen = 32;
assert(keylen == sizeof(scalar));
// open the library
void *libhandle = dlmopen(LM_ID_NEWLM, "/usr/local/lib64/libcrypto.so.3",
RTLD_LAZY | RTLD_DEEPBIND | RTLD_LOCAL);
assert(libhandle != NULL);
// declare the pointer to the function
EVP_PKEY *(*dl_EVP_PKEY_new_raw_private_key)(
int type, ENGINE *e, const unsigned char *key, size_t keylen);
// load from libhandle
if ((dl_EVP_PKEY_new_raw_private_key =
dlsym(libhandle, "EVP_PKEY_new_raw_private_key")) == NULL) {
fprintf(stderr, "dlsym EVP_PKEY_new_raw_private_key: %s\n", dlerror());
}
// create a private key form it.
EVP_PKEY *skey = NULL;
printf("about to create the skey\n");
if (argc > 1) {
// this segfaults somewhere in the library
skey = dl_EVP_PKEY_new_raw_private_key(NID_X25519, NULL, scalar, keylen);
} else {
// this does not
skey = EVP_PKEY_new_raw_private_key(NID_X25519, NULL, scalar, keylen);
}
printf("created the skey\n");
assert(skey != NULL);
dlclose(libhandle);
return 0;
}
Makefile
LIBRARY_PATH=/usr/local/lib64
LIBRARY_NAME=libcrypto.so.3
all: ok fail
a.out: main.c $(LD_LIBRARY_PATH)/$(LIBRARY_NAME)
gcc main.c -ldl -L $(LD_LIBRARY_PATH) -l:$(LIBRARY_NAME)
ok: a.out
@echo sould be ok
LD_LIBRARY_PATH=$(LIBRARY_PATH) ldd ./${^}
LD_LIBRARY_PATH=$(LIBRARY_PATH) ./${^}
@echo ok
@echo
fail: a.out
@echo should segfault
LD_LIBRARY_PATH=$(LIBRARY_PATH) ldd ./${^}
LD_LIBRARY_PATH=$(LIBRARY_PATH) ./${^} dl
@echo wont be able to read this
echo
When executing make you can see that the second call, where we call the dlsymd function, SIGSEGVs. We also see, that the same libcrypto.so.3 is used.
$ make
sould be ok
LD_LIBRARY_PATH=/usr/local/lib64 ldd ./a.out
linux-vdso.so.1 (0x00007ffc0b7b9000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fd36def3000)
libcrypto.so.3 => /usr/local/lib64/libcrypto.so.3 (0x00007fd36dad5000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fd36d909000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fd36df21000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fd36d8e8000)
LD_LIBRARY_PATH=/usr/local/lib64 ./a.out
about to create the skey
created the skey
ok
should segfault
LD_LIBRARY_PATH=/usr/local/lib64 ldd ./a.out
linux-vdso.so.1 (0x00007ffc7bbac000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f9d68d2d000)
libcrypto.so.3 => /usr/local/lib64/libcrypto.so.3 (0x00007f9d6890f000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f9d68743000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f9d68d5b000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f9d68722000)
LD_LIBRARY_PATH=/usr/local/lib64 ./a.out dl
about to create the skey
make: *** [Makefile:18: fail] Segmentation fault (core dumped)
NOTES:
./Configure --api=3.0.0 no-deprecated && make -j and copied the resulting libcrypto.so.3 to /usr/local/lib64/./Configure --api=3.0.0 no-deprecated no-threads it all works flawlessly, which makes me think it is a problem with libpthread+dlsym...coredumpctl debug)Stack trace of thread 1903409:
#0 0x00007fb6d8f8f8ec __pthread_rwlock_rdlock (/usr/lib/libpthread-2.33.so + 0xd8ec)
#1 0x00007fb6d91ba539 CRYPTO_THREAD_read_lock (/usr/local/lib64/libcrypto.so.3 + 0x210539)
#2 0x00007fb6d91a8627 ossl_lib_ctx_get_data (/usr/local/lib64/libcrypto.so.3 + 0x1fe627)
#3 0x00007fb6d91813f4 evp_generic_fetch (/usr/local/lib64/libcrypto.so.3 + 0x1d73f4)
#4 0x00007fb6d918b1c8 EVP_KEYMGMT_fetch (/usr/local/lib64/libcrypto.so.3 + 0x1e11c8)
#5 0x00007fb6d91976f9 EVP_PKEY_CTX_new_from_name (/usr/local/lib64/libcrypto.so.3 + 0x1ed6f9)
#6 0x00007fb6d9192be1 EVP_PKEY_new_raw_private_key (/usr/local/lib64/libcrypto.so.3 + 0x1e8be1)
#7 0x00005575befac2c2 n/a (/home/joel/mwe-dl-libcrypto3/a.out + 0x12c2)
#8 0x00007fb6d9447b25 __libc_start_main (libc.so.6 + 0x27b25)
#9 0x00005575befac0de n/a (/home/joel/mwe-dl-libcrypto3/a.out + 0x10de)
What Am I doing wrong? Is that a dlmopen issue or an openssl issue? (Both seems unlikely to me)
Thanks for providing excellent repro instructions.
What Am I doing wrong?
You are using dlmopen, which is a minefield.
I suspect you are doing this in order to to have several incompatible versions of OpenSSL in a single process. My advice: just don't do it™️.
What's happening... is complicated.
Let's call the first libpthread.so linked into ./a.out P1, and the second copy (which is brought in via dlmopen as a dependency of libcrypto.so) P2. Let's call the dlmopened version of libcrypto C2.
Both P1 and P2 have separate hidden variables __pthread_keys. When C2 calls P2:pthread_key_create, that function looks in P2:__pthread_keys, and discovers that no keys have been used (which is true in P2, but not in P1 -- the loader has already used some keys from P1:__pthread_keys).
So C2 gets an answer from P2:pthread_key_create -- use key==0 (P2 is oblivious to the fact that key==0 has already been used in P1!).
Now C2 calls P2:pthread_getspecific(0), and expects to get a NULL back -- it hasn't called pthread_setspecific(0, ...) yet.
But pthread_getspecific looks in the thread control block, which is unique for the given thread and shared between P1 and P2, and herein lies the disaster: P2 doesn't get a NULL, it gets whatever P1:pthread_setspecific(0, ...) has set previously!
At that point, C2 decides that some other code must have already set up C2's thread-local data appropriately, and proceeds to use that data, with the resulting SIGSEGV.
So who calls P1:pthread_setspecific? It happens here:
Breakpoint 2, __GI___pthread_setspecific (key=0, value=value@entry=0x5555555592a0) at pthread_setspecific.c:33
33 pthread_setspecific.c: No such file or directory.
(gdb) bt
#0 __GI___pthread_setspecific (key=0, value=value@entry=0x5555555592a0) at pthread_setspecific.c:33
#1 0x00007ffff7f9eb3c in _dlerror_run (operate=operate@entry=0x7ffff7f9ee90 <dlmopen_doit>, args=args@entry=0x7fffffffdb60) at dlerror.c:157
#2 0x00007ffff7f9efd9 in __dlmopen (nsid=<optimized out>, file=<optimized out>, mode=<optimized out>) at dlmopen.c:93
#3 0x00005555555551f8 in main ()
And the subsequent call to P2:pthread_getspecific (note the same key==0 being re-used) happens here:
#0 __GI___pthread_getspecific (key=0) at pthread_getspecific.c:30
#1 0x00007ffff77985cd in CRYPTO_THREAD_get_local (key=<optimized out>) at crypto/threads_pthread.c:160
#2 0x00007ffff778a2d2 in get_thread_default_context () at crypto/context.c:166
#3 0x00007ffff778a2ee in get_default_context () at crypto/context.c:171
#4 0x00007ffff778a43b in ossl_lib_ctx_get_concrete (ctx=<optimized out>) at crypto/context.c:278
#5 0x00007ffff778a681 in ossl_lib_ctx_get_data (ctx=<optimized out>, index=index@entry=0, meth=meth@entry=0x7ffff79684e0) at crypto/context.c:356
#6 0x00007ffff776ab6c in get_evp_method_store (libctx=<optimized out>) at crypto/evp/evp_fetch.c:82
#7 0x00007ffff776ab9b in inner_evp_generic_fetch (methdata=methdata@entry=0x7fffffffd9c0, prov=<optimized out>, prov@entry=0x0, operation_id=operation_id@entry=10, name_id=name_id@entry=0, name=0x7ffff78aa0ac "X25519", properties=0x0, new_method=0x7ffff7772e68 <keymgmt_from_algorithm>, up_ref_method=0x7ffff7772d7a <EVP_KEYMGMT_up_ref>,
free_method=0x7ffff7772d88 <EVP_KEYMGMT_free>) at crypto/evp/evp_fetch.c:248
#8 0x00007ffff776b37a in evp_generic_fetch (libctx=<optimized out>, operation_id=operation_id@entry=10, name=<optimized out>, properties=<optimized out>, new_method=new_method@entry=0x7ffff7772e68 <keymgmt_from_algorithm>, up_ref_method=up_ref_method@entry=0x7ffff7772d7a <EVP_KEYMGMT_up_ref>, free_method=0x7ffff7772d88 <EVP_KEYMGMT_free>)
at crypto/evp/evp_fetch.c:372
#9 0x00007ffff77732e8 in EVP_KEYMGMT_fetch (ctx=<optimized out>, algorithm=<optimized out>, properties=<optimized out>) at crypto/evp/keymgmt_meth.c:230
#10 0x00007ffff777d066 in int_ctx_new (libctx=0x0, pkey=pkey@entry=0x0, e=e@entry=0x0, keytype=0x7ffff78aa0ac "X25519", propquery=0x0, id=<optimized out>, id@entry=-1) at crypto/evp/pmeth_lib.c:280
#11 0x00007ffff777d299 in EVP_PKEY_CTX_new_from_name (libctx=<optimized out>, name=<optimized out>, propquery=<optimized out>) at crypto/evp/pmeth_lib.c:368
#12 0x00007ffff7778c70 in new_raw_key_int (libctx=libctx@entry=0x0, strtype=strtype@entry=0x0, propq=propq@entry=0x0, nidtype=1034, e=0x0, key=0x555555556020 <scalar> "\001\002\003\004\005\006\a\b\t\020\021\022\023\024\025\026\027\030\031 !\"#$%&'()012main.c", len=32, key_is_priv=1) at crypto/evp/p_lib.c:406
#13 0x00007ffff7778f3e in EVP_PKEY_new_raw_private_key (type=<optimized out>, e=<optimized out>, priv=<optimized out>, len=<optimized out>) at crypto/evp/p_lib.c:497
#14 0x000055555555529d in main ()
P.S. This only took me 3 hours to debug, and is probably only the first of many problems you are likely to encounter.
P.P.S. Indeed this is only the first problem of many. See this GLIBC bug.