I'm trying to configure OpenSSL so that all OpenSSL applications on my device use a custom engine. I have the following in my openssl.cnf file:
config_diagnostics = 1
openssl_conf = openssl_def
[ openssl_def ]
engines = engine_section
[ engine_section ]
symcrypt = symcrypt_section
[ symcrypt_section ]
engine_id = symcrypt
dynamic_path = /usr/lib/x86_64-linux-gnu/engines-1.1/libsymcryptengine.so
default_algorithms = ALL
init = 1
This works with the OpenSSL application (e.g. when running openssl s_client -tls1_3 -connect google.com:443) and with Nginx. However, it doesn't work with some other applications. For example, I copied the code here which uses the EVP APIs to do a simple symmetric encryption and decryption using AES-CBC. My engine supports AES-CBC, so this should work. But when I build this application (using gcc -o main main.c -lcrypto -lssl) and run it, my engine is not invoked or even loaded. Why not? Is there any way to force all callers of OpenSSL to use my engine without modifying OpenSSL's code? (Or at least all callers that dynamically link OpenSSL and don't explicitly specify their own engine?)
As that page says "Note that this uses the auto-init facility in 1.1.0 [and up]" and the default libcrypto init does not load the standard (or any) config; see the man page for OPENSSL_init_crypto. Either you need to call explicitly with the LOAD_CONFIG flag, or call OPENSSL_init_ssl, or call (first) SSL_CTX_new which would usually be the first libssl routine used in a sensible program and implicitly does OPENSSL_init_ssl.
openssl commandline (in 1.1.0 up) does OPENSSL_init_ssl; I don't know about nginx, but if not it almost certainly starts with SSL_CTX_new.
In older versions (almost all) programs needed to (explicitly) call usually several initialization routines depending on what they were going to do, usually including OPENSSL_add_all_algorithms, and there was a build (compile-time) kludgeoption to have that routine automatically call OPENSSL_config, which is pretty close to what you ask for.