Java sshtools generated EDDSA signature not matching with Python's pycryptome's generated signature

I have a python library that uses pycryptodome library to sign data using Ed25519 algorithm using an openssh format ED25519 private key. The signature then needs to be verified in a Java application using sshtools library with corresponding public key. However the signature verification is failing.

Constraint: It's important to read the private/public keys from files. I cannot change the Python code and/or the keys used.

To debug, I wrote an implementation to generate the signature in Java as well along with validating the python generated signature. However both are coming different.

My Python implementation to sign the data is as follows:

from Crypto.Hash import SHA512
from Crypto.PublicKey import ECC
from Crypto.Signature import eddsa
import base64
import json


def generate_signature_v1(message):
    message = message.replace(" ", "")
    h = SHA512.new(message.encode("utf-8"))
    with open("private", "r") as f:
        key = ECC.import_key(f.read())
    signer = eddsa.new(key, "rfc8032")
    signature = signer.sign(h)
    str_signature = base64.standard_b64encode(signature).decode("utf-8")
    return str_signature

My Java implementation to generate and verify the signature.

import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Base64;
import java.util.HashMap;
import java.util.Map;
import java.io.File;
import java.io.IOException;

import com.google.gson.Gson;
import com.sshtools.common.publickey.InvalidPassphraseException;
import com.sshtools.common.publickey.SshKeyUtils;
import com.sshtools.common.ssh.components.SshKeyPair;
import com.sshtools.common.ssh.components.SshPrivateKey;
import com.sshtools.common.ssh.components.SshPublicKey;

public class StackOverflow {
    private static final Gson gson = new Gson();

    public static boolean verifyV1Signature(String message, String signature) {
        try {
            byte[] messageBytes = message.getBytes(StandardCharsets.UTF_8);
            MessageDigest digest = MessageDigest.getInstance("SHA-512");
            byte[] hash = digest.digest(messageBytes);

            // read public key
            SshPublicKey readPublicKey = SshKeyUtils.getPublicKey(new File("public.pub"));

            // verify signature
            Base64.Decoder decoder = Base64.getDecoder();
            byte[] signatureDecoded = decoder.decode(signature);
            boolean isVerified = readPublicKey.verifySignature(signatureDecoded, hash);
            System.out.println("signature is valid: " + isVerified);

            return isVerified;
        } catch (Exception e) {
            return false;
        }
    }

    public static String generateV1Signature(String message)
            throws NoSuchAlgorithmException, IOException, InvalidPassphraseException {
        byte[] messageBytes = message.getBytes(StandardCharsets.UTF_8);
        MessageDigest digest = MessageDigest.getInstance("SHA-512");
        byte[] hash = digest.digest(messageBytes);

        // create signature
        SshKeyPair readKeyPair = SshKeyUtils.getPrivateKey(new File("private"));
        SshPrivateKey readPrivateKey = readKeyPair.getPrivateKey();
        byte[] signature = readPrivateKey.sign(hash);
        Base64.Encoder encoder = Base64.getEncoder();
        return encoder.encodeToString(signature);
    }

    public static void main(String[] args) {
        Map<String, String> data = new HashMap<>();
        data.put("key", "value");
        String message = gson.toJson(data);
        String pythonSignature = "5Sdt3bIKFbLBhbZ2JLzQP+8MNX6/uzFtxHTkBa/UIpBbjtwKfNu+wfcMHmxksQkmzI5OMhEpY46hVlkM0P5nAA==";
        verifyV1Signature(message, pythonSignature);

        try {
            String javaSignature = generateV1Signature(message);
            System.out.println(javaSignature);
        } catch (NoSuchAlgorithmException | IOException | InvalidPassphraseException e) {
            e.printStackTrace();
        }
    }
}

Running Python code for message json.dumps({"key": "value"}) gives 5Sdt3bIKFbLBhbZ2JLzQP+8MNX6/uzFtxHTkBa/UIpBbjtwKfNu+wfcMHmxksQkmzI5OMhEpY46hVlkM0P5nAA==

Running Java Code gives xHgYq8/nUYOkpbGzCsUkei9Vw0O1/XKoYZlLAbsUPpQF3cTMQ96ROL/ZHSH+cUUNJlmTI2Qb2thAU3kEqvdHBQ== and also verification fails.

The private key looks like -----BEGIN OPENSSH PRIVATE KEY-----<suff>-----END OPENSSH PRIVATE KEY----- and the public key looks like ssh-ed25519 <stuff>

Why signature is not matching? I have also tried bouncycastle and still signature is not matching.

The different signatures arise because different signature algorithms are used unintentionally: In the Python code Ed25519ph is applied, in the Java code Ed25519.
Additionally, in the Java code not the message is signed, but the SHA512 hash of the message.

PyCryptodome supports Ed25519 (PureEdDSA) and Ed25519ph (HashEdDSA), s. here. For Ed25519 the message is to be passed as bytes-like object, for Ed25519ph as hash object.
As Ed25519 is to be used for signing, the message must therefore be passed as bytes-like object, i.e. it must be applied:

...
signature = signer.sign(message.encode("utf-8"))
...

The prepended hardcoded message you mentioned in the comment is a specific feature of Ed25519ph, for more details see RFC 8032, chapters 4, 5.

In addition, the explicit hashing must be removed in the Java code:

byte[] signature = readPrivateKey.sign(messageBytes);

In general, hashing with SHA512 is part of the Ed25519 algorithm and is performed implicitly; the message must not be hashed explicitly with SHA512 (otherwise the SHA512 hash of the message and not the message itself would be signed with Ed25519).

With these changes, both codes produce the same signature for the same input data.