opensslrsabiosuefiedk2

RSA key generation error in RSA_generate_key_ex function call


I'm trying to generate RSA keys using openssl. I've got error on RSA_generate_key_ex and have no idea why the error occures.

I combine ERR_get_error() and ERR_error_string() and I've got next message: error:2506906C:lib(37):func(105):reason(108). Also I found out that 108 error code means RSA_R_DATA_GREATER_THAN_MOD_LEN

I'm trying to generate RSA keys using C code below. For sake of brevity I reduce free calls and error output

RSA* generateRSA()
{
  BIGNUM *bne = BN_new();
  if (bne == NULL)
  {
    return NULL;
  }
  if (BN_set_word(bne, RSA_F4) != 1)
  {
    return NULL;
  }

  RSA *r = RSA_new();
  if (r == NULL)
  {
    return NULL;
  }
  // THERE I'VE GOT ERROR
  if (RSA_generate_key_ex(r, 2048, bne, NULL)!= 1)
  {
    // ERR_get_error() returns 2506906C
    // ERR_error_string() gives me RSA_R_DATA_GREATER_THAN_MOD_LEN
    return NULL;
  }

  return r;
}

The question is what does the error mean and how can I fix it?

Edit: I use OpenSSL 1.1.0e 16 Feb 2017. I use it as part of EDK II Project


Solution

  • I figured out that random generator need to be seeded (openssl versions 1.0.2 and 1.1.0 random generators must be explicitly seeded).

    I check RAND_status(). It returned 0. So the solution is just add RAND_seed() before key generation:

    const void* getSeedBuffer(int num);
    
    RSA* generateRSA()
    {
      RAND_seed(getSeedBuffer(1000), 1000); // don't forget to free memory
      BIGNUM *bne = BN_new();
      if (bne == NULL)
      {
        return NULL;
      }
      if (BN_set_word(bne, RSA_F4) != 1)
      {
        return NULL;
      }
    
      RSA *r = RSA_new();
      if (r == NULL)
      {
        return NULL;
      }
      if (RSA_generate_key_ex(r, 2048, bne, NULL)!= 1)
      {
        return NULL;
      }
    
      return r;
    }