Signing and Verifying rsa signature in polarssl c

I am working on using RSA encryption in an embedded system. for this i'll be using polarssl code.

I had got the encryption working on 128 bit but i am having trouble with the signature part. When i run the code, i get a padding error on the verify (POLARSSL_ERR_RSA_INVALID_PADDING -0x4100)

Below is the code. Problem code is towards the end and the top lines are about the encryption. Still helpful for background info.

int main()
{
    size_t len;
    rsa_context rsa;
    unsigned char rsa_plaintext[PT_LEN];
    unsigned char rsa_decrypted[PT_LEN];
    unsigned char rsa_ciphertext[KEY_LEN];
    unsigned char rsa_hash[PT_LEN];
    unsigned char rsa_sig_out[PT_LEN];
    unsigned char rsa_hash_result[PT_LEN];

rsa_init( &rsa, RSA_PKCS_V15, 0 );
rsa.len = KEY_LEN;

mpi_read_string( &rsa.N , 16, RSA_N  );
mpi_read_string( &rsa.E , 16, RSA_E  );
mpi_read_string( &rsa.D , 16, RSA_D  );
mpi_read_string( &rsa.P , 16, RSA_P  );
mpi_read_string( &rsa.Q , 16, RSA_Q  );
mpi_read_string( &rsa.DP, 16, RSA_DP );
mpi_read_string( &rsa.DQ, 16, RSA_DQ );
mpi_read_string( &rsa.QP, 16, RSA_QP );

// Checking the public and private keys
if( rsa_check_pubkey(  &rsa ) != 0 ||
    rsa_check_privkey( &rsa ) != 0 ) {
    printf( "Public/Private key error! \n" );
    exit(0);
}

memcpy( rsa_plaintext, RSA_PT, PT_LEN );

if( rsa_pkcs1_encrypt( &rsa, &myrand, NULL, RSA_PUBLIC, PT_LEN,
                       rsa_plaintext, rsa_ciphertext ) != 0 ) {
    printf( "Encryption failed! \n" );
    exit(0);
}
if( rsa_pkcs1_decrypt( &rsa, &myrand, NULL, RSA_PRIVATE, &len,
                       rsa_ciphertext, rsa_decrypted,
                       sizeof(rsa_decrypted) ) != 0 ) {
    printf( "Decryption failed! \n" );
    exit(0);
}
if( memcmp( rsa_decrypted, rsa_plaintext, len ) != 0 ) {
    printf( "Compare failed! \n" );
    exit(0);
}
printf("Oh when it all falls down!\n");

// Signing and Verifying message
sha2(rsa_plaintext, len, rsa_hash, 0); //hashing the message 
if (rsa_pkcs1_sign( &rsa, &myrand, NULL, RSA_PRIVATE, SIG_RSA_SHA256, 0, rsa_hash, rsa_sig_out ) != 0) {
    printf( "Signing failed! \n" );
    exit(0);
}
/*
if (rsa_pkcs1_verify( &rsa, NULL, NULL, RSA_PUBLIC, SIG_RSA_SHA256, 0, rsa_sig_out, rsa_hash_result ) != 0) {
    printf( "Verifying signature failed! \n" );
    exit(0);
}
*/
printf("Error Message!:%d \n", rsa_pkcs1_verify( &rsa, NULL, NULL, RSA_PUBLIC,
SIG_RSA_SHA256, 0, rsa_sig_out, rsa_hash_result ));
exit(0);

if( memcmp( rsa_hash, rsa_hash_result, len ) != 0 )
{
    printf( "Signature not verified! \n" );
    exit(0);
}
rsa_free(&rsa);

return 0;

}

Anybody know how i can solve this and go on ahead. Please let me know. Thanks I am running this on MinGw gcc compiler for windows. The rsa code has dependencies on bignum, md and sha2.

The reason why the hash fails is because you didn't fill rsa_hash before signing or rsa_hash_result before verification.

rsa_pkcs1_sign() and rsa_pkcs1_verify() sign and verify the hash presented. They do not make the hash as they do not know the data. (i.e. rsa_plaintext and rsa_ciphertext never enter the sign or verify functions).

So before you call rsa_pkcs1_sign() you should run sha256(rsa_plaintext, rsa_hash); or sha256(rsa_ciphertext, rsa_hash); (depending on how your 'protocol' works).

Then before verify, you run sha256(XXX, rsa_hash_result); and provide that value to rsa_pkcs1_verify() so that it can actually verify your hash.