How can I add some new functionalities to the OpenSSL API in order to use the GPU for some cryptographic algorithms (AES, RSA, etc) written by me? I'd want, for example, to use the command openssl -engine cuda_engine genrsa -out rsa.key 1024 and the OpenSSL to use my genrsa CUDA code instead of the original code. I have to mention that I want to do this on Windows 7 SP1, 64-bit. Even a dummy solution would be helpful because I need to understand the mechanism.
Heres a couple of in-depth posts written by an OpenSSL dev about building an engine.
Lesson 1: A Minimum Useless Engine
Lesson 2: An Example MD5 Engine
I would also recommend looking through the engines that are in the OpenSSL source tree.
You only need to recompile openssl if you dont have dynamic engine support enabled. Additonial info OpenSSL Wiki
For openssl app/library add to openssl.cnf.
openssl_conf = openssl_def
[openssl_def]
engines = engines_section
[engines_section]
silly = silly_section
[silly_section]
engine_id = silly
dynamic_path = /home/workingdir/silly_engine.dll
init = 0