Convert existing ed25519 private key file in openssl "private" format into ssh/ssh-keygen format using python3

I have an existing ed25519 keypair generated by openssl stored in files. The key files are in openssl "private" format, e.g. (these are sample keys; no secrets):

This is not compatible with ssh and ssh-keygen. Yes, ssh-keygen -t ed25519 will create a "traditional" key representation that works e.g.:

but I cannot generate new keys; I must use the existing openssl-generated key and convert it to something I can use in ssh.

It is my understanding that in fact there is no direct conversion using command line utils. I am happy to use python3 and the ecdsa, cryptography, and other modules to perform this conversion. I have tried all sorts of things including:

import base64
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519

private_key_data = 'MC4CAQAwBQYDK2VwBCIEIH7sjlQYpBCnodJqPqYS2441L4wOOqyfLoc/SzTTC1h8' # We will deal with reading the file and dropping header/footer later
# Decode the base64 data                                                                    
private_key_bytes = base64.b64decode(private_key_data)

# Create an Ed25519 private key object                                                      
private_key = ed25519.Ed25519PrivateKey.from_private_bytes(private_key_bytes)

# Serialize the private key to PEM format                                                   
private_key_pem = private_key.private_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PrivateFormat.PKCS8,
    encryption_algorithm=serialization.NoEncryption()
)
# Get the private key as a string                                                           
private_key_pem_str = private_key_pem.decode('utf-8')

print(private_key_pem_str)

but every variation ends up with something similar to:

ValueError: An Ed25519 private key is 32 bytes long

I am pretty much stuck trying to create an internal object from this openssl generated private format. Any suggestions?

Solution

The posted private Ed25519 key has PKCS#8 format, the posted public Ed25519 key has X.509/SPKI format. Both keys are PEM encoded.
Indeed, the pyca/cryptography library supports conversion to OpenSSH format as follows:

from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.serialization import load_pem_private_key, load_pem_public_key

# Private key
pkcs8Pem = b"""-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEIH7sjlQYpBCnodJqPqYS2441L4wOOqyfLoc/SzTTC1h8
-----END PRIVATE KEY-----"""

privateKey = load_pem_private_key(pkcs8Pem, password=None)
privateOpenSshPem = privateKey.private_bytes(
   encoding=serialization.Encoding.PEM,
   format=serialization.PrivateFormat.OpenSSH,
   encryption_algorithm=serialization.NoEncryption()
)
print("private key:\n", privateOpenSshPem.decode())

# Public key
x509SpkiPem = b"""-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEAsXTnNvb1du7vk97WGRlaieCen309UgxWB8wrBCDkw0M=
-----END PUBLIC KEY-----"""

publicKey = load_pem_public_key(x509SpkiPem)
publicOpenSsh = publicKey.public_bytes(
   encoding=serialization.Encoding.OpenSSH,
   format=serialization.PublicFormat.OpenSSH
)
print("public key:\n", publicOpenSsh.decode())

with the output:

private key:
 -----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZWQyNTUx
OQAAACAsGlh6oOzQbqjPJJhZTH330iDsRuPpJDvLpN+Fj5oj5QAAAIg7VSakO1UmpAAAAAtzc2gt
ZWQyNTUxOQAAACAsGlh6oOzQbqjPJJhZTH330iDsRuPpJDvLpN+Fj5oj5QAAAEB+7I5UGKQQp6HS
aj6mEtuONS+MDjqsny6HP0s00wtYfCwaWHqg7NBuqM8kmFlMfffSIOxG4+kkO8uk34WPmiPlAAAA
AAECAwQF
-----END OPENSSH PRIVATE KEY-----

public key:
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILF05zb29Xbu75Pe1hkZWongnp99PVIMVgfMKwQg5MND

Some info about the formats: The posted PKCS#8 and X.509/SPKI keys are PEM encoded. If in the body the line breaks are removed and a Base64 decoding is performed, this results in the DER encoded key. DER is an encoding for ASN.1, an interface definition language.
To display the ASN.1, it is best to load the key into an ASN.1 parser, e.g. https://lapo.it/asn1js/, e.g. PEM encoded or as Base64 encoded DER, e.g. for the private key:

On the left is the ASN.1 key, on the right is the DER encoding. For Ed25519, the last 32 bytes are the raw private key (and in the case of the public key, the raw public key).
In contrast, no ASN.1 is used for the private key in the OpenSSH format. A detailed description of this format can be found e.g. here, section OpenSSH Private Keys.

EDIT - Regarding your comment:

The problem is probably not with the transformation: If you use pyca/cryptography to convert a private and public key from OpenSSH format to OpenSSL format (PKCS#8, X.509/SPKI) and back to OpenSSH format, you get the same OpenSSH keys (except, of course, the OpenSSH comment, which is lost in this reconstruction, since such a comment is not stored in the OpenSSL format).

Contrary, a more likely cause for the problem would be non-associated keys: ssh-keygen -t ed25519 creates a key pair, i.e. a private and public key that belong to each other. However, the keys you posted are not associated, so they belong to two different key pairs. This can be verified by deriving the public key from the private key with:

openssl pkey -in <path to private key> -pubout

which results for the posted private key in:

-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEALBpYeqDs0G6ozySYWUx999Ig7Ebj6SQ7y6TfhY+aI+U=
-----END PUBLIC KEY-----

and is not identical to the posted public key. It is possible that this is also true for your real keys. If so, this could be an explanation for your approach failing.

To check whether the transformation is the cause of the problem, you could convert a key pair in OpenSSH format (which you created with ssh-keygen -t ed25519 and checked that it works) to OpenSSL format (PKCS#8, X.509/SPKI) and back to OpenSSH format using pyca/cryptography. Then you could check with these re-transformed OpenSSH keys if it works. If so, the problem is not with the transformation, but most likely with your real keys.
Alternatively, you can post a working (but non-productive) key pair in OpenSSH format, I'll do the transformation, and you check the back-transformed OpenSSH keys on your end.