How to encrypt data in node.js using node-rsa and decrypt encrypted data in android?

Node.js code for encryption

   function encrypt(msg) {
        try {
            const public_key = fs.readFileSync(path.join(__dirname, 'PublicKey.pem'), 'utf8');
            const encryptStr = crypto.publicEncrypt({
                key: public_key,
                padding: crypto.constants.RSA_PKCS1_PADDING
            }, Buffer.from(msg,'utf8'));
            let encryptString = encryptStr.toString('hex');
            return encryptString;
        } catch (e) {
            console.log(e);
            return false;
        }
    }

Android Code for decryption

  public static String decryptStringWithPrivateKey(String s, String keyFilename)  throws Exception {
        Cipher cipher = Cipher.getInstance("RSA");
        PrivateKey pkey = readPrivateKeyFromPem(keyFilename);

        cipher.init(Cipher.DECRYPT_MODE, pkey);
        String dec = new String(cipher.doFinal(Base64.getDecoder().decode(s)), "UTF-8");

        return dec;
    }


   public static PrivateKey readPrivateKeyFromPem(String keyFilename) throws Exception {
        byte[] keyBytes = Files.readAllBytes(new File(keyFilename).toPath());
        String keyString = new String(keyBytes);

        if (keyString.contains("BEGIN PRIVATE KEY")) {
            return readPrivateKeyFromPem_PKCS8(keyFilename);
        }
        else if(keyString.contains("BEGIN RSA PRIVATE KEY")){
            return readPrivateKeyFromPem_PKCS1(keyFilename);
        }
     
        throw new Exception("Unknown private key format in "+keyFilename);
    }

I want to encrypt api on server side and decrypt the data on android device to avoid unauthorized access.

while decryption on android "Input byte array has incorrect ending byte at 172" this one error are thrown.

Solution

In the NodeJS code the cipher text is hex encoded and in the Java code it is Base64 decoded. This must be made consistent, either Base64 or hex encoding on both sides.

Also, in the Java code, when instantiating the cipher, only the algorithm is specified. This results in a default padding being used, which e.g. on my machine (API 28, Android 9 Pie) corresponds to RSA/ECB/NoPadding. This is not compatible with the NodeJS side, which uses PKCS#1 v1.5 padding (besides that RSA without padding, so called textbook RSA, is insecure). Therefore, padding must also be specified in the cipher instantiation with RSA/ECB/PKCS1Padding.

Using a private key in PKCS#8 format and an own implementation of readPrivateKeyFromPem_PKCS8() both codes work on my machine if the two bugs mentioned above are fixed. However, the methods not posted could also contain flaws.