I'm updating some code that I didn't touch for several years, and to my surprise it produces a runtime error. I'm very sure it used to work just fine.

Reproducer

#include <openssl/evp.h>
#include <openssl/err.h>

/* Compile with
 * gcc filename.c -lcrypto
 */

void handleErrors(void)
{
    ERR_print_errors_fp(stderr);
    abort();
}

int main(int argc, char** argv)
{
    EVP_CIPHER_CTX *ctx;
    int rc;

    /* Create context */
    if ( !(ctx = EVP_CIPHER_CTX_new()) )
        handleErrors();

    // Deactivate padding. 0 stands for "false", not for padding length.
    if ( 1 != (rc = EVP_CIPHER_CTX_set_padding(ctx, 0))) {
        printf("rc = %i\n", rc);
        handleErrors();
    }

    // EVP_EncryptInit_ex() to be called after
}

produces

$ ./a.out
rc = 0
Aborted (core dumped)

Questions

• Why does EVP_CIPHER_CTX_set_padding() return 0? It should always return 1.
• Why doesn't ERR_print_errors_fp() print an error?

Context

I'm using OpenSSL 3.0.17-1~deb12u2 on Debian 12 "bookworm". (A.k.a. "oldstable" as of late 2025.)

As comments have pointed out, OpenSSL EVP_CIPHER_CTX_set_padding not working contained the solution to this question. I even read this post but managed to misunderstand the answer.

The problem was that EVP_CIPHER_CTX_set_padding() got called before EVP_EncryptInit_ex(). Switching the order of these function makes EVP_CIPHER_CTX_set_padding() not emit an error.

Perhaps this error mode got added to make the problem in OpenSSL EVP_CIPHER_CTX_set_padding not working surface quicker. Don't know, but it would explain why this code worked fine many years ago. Don't know for sure, though.

#include <openssl/evp.h>
#include <openssl/err.h>

/* Compile with
 * gcc error.c -lcrypto -o
 */

void handleErrors(void)
{
    ERR_print_errors_fp(stderr);
    abort();
}

int main(int argc, char** argv)
{
    EVP_CIPHER_CTX *ctx;
    int rc;

    /* Create and initialise the context */
    if (!(ctx = EVP_CIPHER_CTX_new()))
        handleErrors();

    // Set engine implementation
    if ( 1 != EVP_EncryptInit_ex(ctx, EVP_aes_128_gcm(), NULL, NULL, NULL)) {
        printf("EVP_EncryptInit_ex() failed");
        handleErrors();
    }

    // Deactivate padding. 0 stands for "false", not for padding length.
    if ( 1 != (rc = EVP_CIPHER_CTX_set_padding(ctx, 0))) {
        printf("rc = %i\n", rc);
        handleErrors();
    }
}