python kotlin encryption cryptography rsa

Can't align RSA encryption in Python and Kotlin

I would like to add RSA encryption in my server (Python FastAPI) and my Android app.
But the encryption didn't work as the way I expected.

I already have AES-GCM encryption/decryption working between my Python and Kotlin code. However, my RSA attempts in Python and Kotlin won't interoperate with each other.

The Python (server) RSA code can decrypt what the Python RSA code encrypts, and the Kotlin (app) RSA code can decrypt what the Kotlin RSA code encrypts.

I used the cryptography module in Python and the native cryptography in Kotlin.

Here is my Python file.

import os, base64, re
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.primitives import serialization, hashes
from cryptography.hazmat.backends import default_backend

if os.path.exists("private.key") and os.path.exists("public.key"):
    print("Loading existing keys")
    with open("private.key", "rb") as pkf, open("public.key", "rb") as kf:
        pk = serialization.load_der_private_key(pkf.read(), None, default_backend())
        k = serialization.load_der_public_key(kf.read(), default_backend())
else:
    print("Generating new keys")
    pk = rsa.generate_private_key(65537, 4096, default_backend())
    pkb = pk.private_bytes(
        serialization.Encoding.DER,
        serialization.PrivateFormat.PKCS8,
        serialization.NoEncryption()
    )
    k = pk.public_key()
    kb = k.public_bytes(
        serialization.Encoding.DER,
        serialization.PublicFormat.SubjectPublicKeyInfo
    )
    
    with open("private.key", "wb") as pkf, open("public.key", "wb") as f:
        pkf.write(pkb)
        f.write(kb)

def enc(text):
    return base64.b64encode(k.encrypt(
        text.encode(),
        padding.OAEP(
            mgf=padding.MGF1(hashes.SHA512()),
            algorithm=hashes.SHA512(),
            label=None
        )
    )).decode()

def dec(ciphertext):
    return pk.decrypt(
        base64.b64decode(ciphertext),
        padding.OAEP(
            mgf=padding.MGF1(hashes.SHA512()),
            algorithm=hashes.SHA512(),
            label=None
        )
    ).decode()

while (inp:= input("RSA: ")) != "":
    if re.match(f"^enc", inp):
        print("Encrypt", enc(inp[3:].strip()))
    if re.match(f"dec", inp):
        print("Decrypt", dec(inp[3:].strip()))

And here is my Kotlin file.

import java.security.KeyPair
import java.security.KeyPairGenerator
import java.security.SecureRandom
import java.security.KeyFactory
import java.security.PublicKey
import java.security.spec.X509EncodedKeySpec
import java.security.spec.PKCS8EncodedKeySpec
import javax.crypto.Cipher
import javax.crypto.SecretKey
import javax.crypto.KeyGenerator
import javax.crypto.spec.SecretKeySpec
import javax.crypto.spec.GCMParameterSpec
import java.util.Base64
import java.io.File
import java.nio.file.Files

fun main(){
    print("Text: ")
    val encrypted = rsaEncrypt(File("public.key"), readLine().toString())
    println(encrypted)
    print("Ciphertext: ")
    println("Decrypted: ${rsaDecrypt(File("private.key"), readLine().toString())}")
}

fun bEncode(data: ByteArray) = Base64.getEncoder().encodeToString(data)

fun bDecode(string: String) = Base64.getDecoder().decode(string)

fun rsaEncrypt(keyFile: File, text: String): String {
    val k = KeyFactory.getInstance("RSA").generatePublic(X509EncodedKeySpec(keyFile.readBytes()))
    val c = Cipher.getInstance("RSA/ECB/OAEPwithSHA-512andMGF1Padding")
    c.init(Cipher.ENCRYPT_MODE, k)
    return bEncode(c.doFinal(text.toByteArray()))
}

fun rsaDecrypt(keyFile: File, ciphertext: String): String {
    val k = KeyFactory.getInstance("RSA").generatePrivate(PKCS8EncodedKeySpec(keyFile.readBytes()))
    val c = Cipher.getInstance("RSA/ECB/OAEPwithSHA-512andMGF1Padding")
    c.init(Cipher.DECRYPT_MODE, k)
    return c.doFinal(bDecode(ciphertext)).decodeToString()
}

According my file, first I run my Python file then my Kotlin file.
Here is the error in Kotlin while trying to decrypt Python encrypted data.

Exception in thread "main" javax.crypto.BadPaddingException: Padding error in decryption at java.base/com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:389) at java.base/com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:425) at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2205) at TestKt.rsaDecrypt(test.kt:58) at TestKt.main(test.kt:22) at TestKt.main(test.kt)

Here is the error in Python while trying to decrypt Kotlin encrypted data.

Traceback (most recent call last): File "C:\Users\MYUSERNAME\Projects\MyServer\test.py", line 53, in <module> print("Decrypt", dec(inp[3:].strip())) ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\MYUSERNAME\Projects\MyServer\test.py", line 40, in dec return pk.decrypt( ^^^^^^^^^^^ ValueError: Decryption failed

What wrong with my code?

Here is the execution of the codes. (I added print/println of type and size of encrypted data of both Python and Kotlin. It look normal.)

S C:\Users\USERNAME\Projects\MyServer> python test.py && kotlinc test.kt -include-runtime -d app.jar && kotlin app.jar
Loading existing keys
RSA: enc Hello
<class 'bytes'>
512
Encrypt sMeiZvcV5aHIFsW7+pOhcAwi/Jk1Fc0JevLM9U5LK+005vFgxUgVehLgwgMG/9TJ7wPOVYI51Cnrhc4aLBfG6gawWoc+NGO8WfGA7HF0xFgdDreAIzRMGOsR7AkoVBsBz7HbjnQ+3Nz1AOUHAUbrVLLizAvCjQAB0xbvv/YtQWRXrZOCEbF3+RFF9GOWQO8Lk8Wd/KvnD6pyASw8QvGqEJXL8aWZH0P8FDYUEcwjrz4qOgZDBce8qdiVf8h2FItm/AzeWvBuMai0FSh4SYjYMW2Ld8BGACsBIhYtpXG62WVULvXHK5oGQ5mpJz62WKCauDeqGdugeP9LVDA0k3Nd645hU6WCCA5o4VrKhz4WR4uN6ND42qNCIIE9LLMyHBGcZ+bOf0pGraLqbOHOYkD4bLD3j32ILT88WpC31KhFjvdCxBv81IXdl0/6f9U+q4d6ZZ9OiWjDdINemFVxgis9d1GzuTYzQ2i3MGudHfnFAMKjZWW8IJIfaIfAiMob/WjO2VTGvn0eFaUviszFaAfzTcoe0AnhyUpEEgu1KUcJk/CUoSnKYRX62T2wgmHKgfavdeKYmLPKugORRcCj9hlkke2rlTva0EP9970UycOwJGb92CwxPn68C8wA/jmKxoj7U2CDUArTwqV/dzvQ/gosfSSF/bWB9ZSoAi2//JVlk+g=
RSA: dec sMeiZvcV5aHIFsW7+pOhcAwi/Jk1Fc0JevLM9U5LK+005vFgxUgVehLgwgMG/9TJ7wPOVYI51Cnrhc4aLBfG6gawWoc+NGO8WfGA7HF0xFgdDreAIzRMGOsR7AkoVBsBz7HbjnQ+3Nz1AOUHAUbrVLLizAvCjQAB0xbvv/YtQWRXrZOCEbF3+RFF9GOWQO8Lk8Wd/KvnD6pyASw8QvGqEJXL8aWZH0P8FDYUEcwjrz4qOgZDBce8qdiVf8h2FItm/AzeWvBuMai0FSh4SYjYMW2Ld8BGACsBIhYtpXG62WVULvXHK5oGQ5mpJz62WKCauDeqGdugeP9LVDA0k3Nd645hU6WCCA5o4VrKhz4WR4uN6ND42qNCIIE9LLMyHBGcZ+bOf0pGraLqbOHOYkD4bLD3j32ILT88WpC31KhFjvdCxBv81IXdl0/6f9U+q4d6ZZ9OiWjDdINemFVxgis9d1GzuTYzQ2i3MGudHfnFAMKjZWW8IJIfaIfAiMob/WjO2VTGvn0eFaUviszFaAfzTcoe0AnhyUpEEgu1KUcJk/CUoSnKYRX62T2wgmHKgfavdeKYmLPKugORRcCj9hlkke2rlTva0EP9970UycOwJGb92CwxPn68C8wA/jmKxoj7U2CDUArTwqV/dzvQ/gosfSSF/bWB9ZSoAi2//JVlk+g=
Decrypt Hello
RSA:
Text: Hello
ByteArray
512
XgLJElQ46cqiVmwqs/j2y8NptEU9ciAnQFtuuh2U+4m1PbqatnzVrB//G6NXrH2hYbclj9GyHMAzRh3f9LlGozQl7FgvmGp6F38DD2j2ktVChAMLCWtw7GMkJlyJiINFdiYL1IV1EI/+DpQYzeE94tyQDaev053GAW+aFxymWTVSZ1uw97XJGCd1V2RRyzzPKirG/BTxKS+j9iqsGUCd+7SVghzhWlzZYgaPj+7t0by45SURvrtTdaD6Ni1FIwROPOzTEE6ryFbK0tPWECN1jnQI/1qylldE/N/Awqc0ORsj9wCITh80D2ibfZyr/AN5I+QOU3y4pfX76C6xAvsFt1Avsk15sktsNfrBC3+OyNB1PWiKmeXcxly2rx4PswpL56WEOkcrhF6GcyctfU5gJNDQ/CsND+u3/JNtPHtPdmAhwoY0UWeUeZWIp0yR42fbrLBgFzPodahHNXopMhBlGO84yzO6WNGbm/0lBnVSm6PAw6ti5N0gSC6eNq/odSQQYThVOvxJB9ETPbzX8CaV8GmQUbiGVzR8P5HOvYK4ctyIJha/X1rP4CpbKXsKVLYWuvGKJ1T7fG6/69/pKLJC3MkurpM2OVlb5UgV5W1A4fCQROQ9yTt4jRUllgmORvaUrOZO62rHKnFmA6cxRXou36t61HKYAmQ3ahq/ErBQyuI=
Ciphertext: XgLJElQ46cqiVmwqs/j2y8NptEU9ciAnQFtuuh2U+4m1PbqatnzVrB//G6NXrH2hYbclj9GyHMAzRh3f9LlGozQl7FgvmGp6F38DD2j2ktVChAMLCWtw7GMkJlyJiINFdiYL1IV1EI/+DpQYzeE94tyQDaev053GAW+aFxymWTVSZ1uw97XJGCd1V2RRyzzPKirG/BTxKS+j9iqsGUCd+7SVghzhWlzZYgaPj+7t0by45SURvrtTdaD6Ni1FIwROPOzTEE6ryFbK0tPWECN1jnQI/1qylldE/N/Awqc0ORsj9wCITh80D2ibfZyr/AN5I+QOU3y4pfX76C6xAvsFt1Avsk15sktsNfrBC3+OyNB1PWiKmeXcxly2rx4PswpL56WEOkcrhF6GcyctfU5gJNDQ/CsND+u3/JNtPHtPdmAhwoY0UWeUeZWIp0yR42fbrLBgFzPodahHNXopMhBlGO84yzO6WNGbm/0lBnVSm6PAw6ti5N0gSC6eNq/odSQQYThVOvxJB9ETPbzX8CaV8GmQUbiGVzR8P5HOvYK4ctyIJha/X1rP4CpbKXsKVLYWuvGKJ1T7fG6/69/pKLJC3MkurpM2OVlb5UgV5W1A4fCQROQ9yTt4jRUllgmORvaUrOZO62rHKnFmA6cxRXou36t61HKYAmQ3ahq/ErBQyuI=
Decrypted: Hello
PS C:\Users\USERNAME\Projects\MyServer> kotlin app.jar
Text: Hi
ByteArray
512
hS4WeVI6Vuh/bdyiDMvBtyYRBfRWHkSJsrmSe4ivq2Xov+3jRXMqxajPiJ6gH9qIB4U4pFa2sy0gdjM+fQtzqv86JPAvbGjO+fFYQhunvVwJwIlR/3fwUJRSkzaYNTR+SsZ4mTJKB45bwiurZuOQFfQ8e72cD+UBkAPJmFx5FftazKIis0ehwdJ+G3ZAnTwzPM82dqIQCMfb6JeeT2F7BLhdc06pKQXiWPDRytkDSHfOXf0SWJJ5Kwbgy6G10X68VRND/DsS3CXrxr2us8/9IrZZw2sCkE72MB38/sUSB5lwYeia5Rf3OxKzq1VQrHcI8obdIErVg7NNsKfSMdxA2pIHzZfFnu9by7gOw+hy95ntrdjE9RE9IcEt5ylZ0ZFmvCjnAJ1/TbqhZBN70blxMJDrqiYoM0x/M3EjdTuodlJ+deNZI84mfAmCmzDTj2umFT6X0HKFJPQVaZYDIyDq+EJueYOdS3xKHfE2ycWkQSPh9lmebA1GNuv6EgGMMiEkw86mFvAHe9jgSkYkLI7+qMjARuJLTYXKINihVRDyyzcMAq7eDuFDoVWavNu/cw5AXJO8TZzSGTFtoLgfXpck1BKfnrb3IEe01pqADJArX3hgCB22xLEZ9zktj71L1T/gHrEfMM4bnJkpyvVEDm9kajMIwFZa3hq8FztoDkNzbFI=
Ciphertext: sMeiZvcV5aHIFsW7+pOhcAwi/Jk1Fc0JevLM9U5LK+005vFgxUgVehLgwgMG/9TJ7wPOVYI51Cnrhc4aLBfG6gawWoc+NGO8WfGA7HF0xFgdDreAIzRMGOsR7AkoVBsBz7HbjnQ+3Nz1AOUHAUbrVLLizAvCjQAB0xbvv/YtQWRXrZOCEbF3+RFF9GOWQO8Lk8Wd/KvnD6pyASw8QvGqEJXL8aWZH0P8FDYUEcwjrz4qOgZDBce8qdiVf8h2FItm/AzeWvBuMai0FSh4SYjYMW2Ld8BGACsBIhYtpXG62WVULvXHK5oGQ5mpJz62WKCauDeqGdugeP9LVDA0k3Nd645hU6WCCA5o4VrKhz4WR4uN6ND42qNCIIE9LLMyHBGcZ+bOf0pGraLqbOHOYkD4bLD3j32ILT88WpC31KhFjvdCxBv81IXdl0/6f9U+q4d6ZZ9OiWjDdINemFVxgis9d1GzuTYzQ2i3MGudHfnFAMKjZWW8IJIfaIfAiMob/WjO2VTGvn0eFaUviszFaAfzTcoe0AnhyUpEEgu1KUcJk/CUoSnKYRX62T2wgmHKgfavdeKYmLPKugORRcCj9hlkke2rlTva0EP9970UycOwJGb92CwxPn68C8wA/jmKxoj7U2CDUArTwqV/dzvQ/gosfSSF/bWB9ZSoAi2//JVlk+g=
Exception in thread "main" javax.crypto.BadPaddingException: Padding error in decryption
        at java.base/com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:389)
        at java.base/com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:425)
        at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2205)
        at TestKt.rsaDecrypt(test.kt:59)
        at TestKt.main(test.kt:22)
        at TestKt.main(test.kt)
PS C:\Users\USERNAME\Projects\MyServer> python test.py
Loading existing keys
RSA: dec XgLJElQ46cqiVmwqs/j2y8NptEU9ciAnQFtuuh2U+4m1PbqatnzVrB//G6NXrH2hYbclj9GyHMAzRh3f9LlGozQl7FgvmGp6F38DD2j2ktVChAMLCWtw7GMkJlyJiINFdiYL1IV1EI/+DpQYzeE94tyQDaev053GAW+aFxymWTVSZ1uw97XJGCd1V2RRyzzPKirG/BTxKS+j9iqsGUCd+7SVghzhWlzZYgaPj+7t0by45SURvrtTdaD6Ni1FIwROPOzTEE6ryFbK0tPWECN1jnQI/1qylldE/N/Awqc0ORsj9wCITh80D2ibfZyr/AN5I+QOU3y4pfX76C6xAvsFt1Avsk15sktsNfrBC3+OyNB1PWiKmeXcxly2rx4PswpL56WEOkcrhF6GcyctfU5gJNDQ/CsND+u3/JNtPHtPdmAhwoY0UWeUeZWIp0yR42fbrLBgFzPodahHNXopMhBlGO84yzO6WNGbm/0lBnVSm6PAw6ti5N0gSC6eNq/odSQQYThVOvxJB9ETPbzX8CaV8GmQUbiGVzR8P5HOvYK4ctyIJha/X1rP4CpbKXsKVLYWuvGKJ1T7fG6/69/pKLJC3MkurpM2OVlb5UgV5W1A4fCQROQ9yTt4jRUllgmORvaUrOZO62rHKnFmA6cxRXou36t61HKYAmQ3ahq/ErBQyuI=
Traceback (most recent call last):
  File "C:\Users\USERNAME\Projects\MyServer\test.py", line 56, in <module>
    print("Decrypt", dec(inp[3:].strip()))
                     ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\Projects\MyServer\test.py", line 43, in dec
    return pk.decrypt(
           ^^^^^^^^^^^
ValueError: Decryption failed

Solution

I've found that it's best to avoid defaults when writing cryptography code. The problem is that it's hard to know when you're getting defaults because there are no warnings. In this case, the OAEP scheme has a few parameters that you should always specify. These can be set on the Java/Kotlin side with the OAEPParameterSpec. So, you're Kotlin code should look something like

fun rsaEncrypt(keyFile: File, text: String): String {
    val k = KeyFactory.getInstance("RSA").generatePublic(X509EncodedKeySpec(keyFile.readBytes()))
    val c = Cipher.getInstance("RSA/ECB/OAEPwithSHA-512andMGF1Padding")
    val oaepSpec = OAEPParameterSpec("SHA-512", "MGF1", MGF1ParameterSpec("SHA-512"), PSource.PSpecified.DEFAULT)
    c.init(Cipher.ENCRYPT_MODE, k, oaepSpec)
    return bEncode(c.doFinal(text.toByteArray()))
}

fun rsaDecrypt(keyFile: File, ciphertext: String): String {
    val k = KeyFactory.getInstance("RSA").generatePrivate(PKCS8EncodedKeySpec(keyFile.readBytes()))
    val c = Cipher.getInstance("RSA/ECB/OAEPwithSHA-512andMGF1Padding")
    val oaepSpec = OAEPParameterSpec("SHA-512", "MGF1", MGF1ParameterSpec("SHA-512"), PSource.PSpecified.DEFAULT)
    c.init(Cipher.DECRYPT_MODE, k, oaepSpec)
    return c.doFinal(bDecode(ciphertext)).decodeToString()
}

If I remember correctly, the default on the Java side for the MGF1 hash is SHA-1, so that's why it didn't work.