OpenSSL apparently producing invalid key pairs for ed25519

I'm trying to generate a self-signed certificate for ed25519 keys using openSSL, using the following command:

openssl req -new -newkey ed25519 -noenc -config root.cnf -section root_req -out RootCA.csr -keyout RootCA.key -passout file:password.txt -batch

These are the relevant config sections (I doubt they matter here):

[root_req]
default_bits            = 256
encrypt_key             = no
default_md              = sha256
utf8                    = yes
string_mask             = utf8only
prompt                  = no
distinguished_name      = root_dn
req_extensions          = root_ext

[root_ext]
basicConstraints        = critical,CA:true
keyUsage                = critical,keyCertSign
subjectKeyIdentifier    = hash

This generates a request with the public key:

BFDC8F370A5B5CA66E5FA55A6CFBC6D10963B2D6EFFCB86C006E08F64A0654D8

And the resulting signed cert agrees:

Subject Public Key Info:
    Public Key Algorithm: ED25519
        ED25519 Public-Key:
        pub:
            bf:dc:8f:37:0a:5b:5c:a6:6e:5f:a5:5a:6c:fb:c6:
            d1:09:63:b2:d6:ef:fc:b8:6c:00:6e:08:f6:4a:06:
            54:d8

The req command also generates a RootCA.key file:

-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEIJkwKy4DxV2Ve6o4xkvtkDIbU1vcKGSjKrGL7W8RxxWd
-----END PRIVATE KEY-----

I decode that to 48 bytes, first 16 being the algorithm marker 302e020100300506032b657004220420, and the private key being:

99302B2E03C55D957BAA38C64BED90321B535BDC2864A32AB18BED6F11C7159D

I then try to verify the keys using NaCl.net, which also implements ed25519, and here the trouble start.

Curve25519XSalsa20Poly1305.Encrypt and .TryDecrypt fails when using this key set.

Furthermore, when using Curve25519.ScalarMultiplicationBase to reconstruct the public key from the private key, they don't match. NaCl.net claims the public key associated with above private key 99302B... should instead be:

5089A9E4FBFF67D9419713FC0267C1540091E0126607BF903F4996E03EC79F29

And this key pair works, they succesfully sign a signature.

So... why do the keys output by openSSL not seem like they match? Have I screwed up somewhere in request generation, or is there some formatting difference between openSSL and NaCl.net that's tricking me?

With the OpenSSL statement you generate keys for Ed25519, which is intended for signing.

The NaCl.net library (respectively the type Curve25519XSalsa20Poly1305) however implements public-key authenticated encryption with X25519 for the key agreement.

The keys for Ed25519 or X25519 are not simply interchangeable, but can be converted.

So for a verification of your OpenSSL keys you need a .NET library for Ed25519, e.g. Sodium.Core. This can be used to derive the public key from the secret key:

using Sodium;

byte[] secretEd25519 = Convert.FromHexString("99302B2E03C55D957BAA38C64BED90321B535BDC2864A32AB18BED6F11C7159D");
KeyPair keyPair = PublicKeyAuth.GenerateKeyPair(secretEd25519);
Console.WriteLine(Convert.ToHexString(keyPair.PublicKey)); // BFDC8F370A5B5CA66E5FA55A6CFBC6D10963B2D6EFFCB86C006E08F64A0654D8

The public key derived from the secret key is identical to the public key calculated by OpenSSL.

---

Alternatively, the Ed25519 can be converted to X25519 keys and the consistency of the X25519 keys can be checked:

using Sodium;

byte[] secretEd25519 = Convert.FromHexString("99302B2E03C55D957BAA38C64BED90321B535BDC2864A32AB18BED6F11C7159D");
byte[] publicEd25519 = Convert.FromHexString("BFDC8F370A5B5CA66E5FA55A6CFBC6D10963B2D6EFFCB86C006E08F64A0654D8");

byte[] secretX25519 = PublicKeyAuth.ConvertEd25519SecretKeyToCurve25519SecretKey(secretEd25519);
byte[] publicX25519 = PublicKeyAuth.ConvertEd25519PublicKeyToCurve25519PublicKey(publicEd25519);
Console.WriteLine(Convert.ToHexString(secretX25519));      // C01A8724CB3C3EB03C1B015C1B91DEF56136391088EB5D1E60CB3AB56068D364
Console.WriteLine(Convert.ToHexString(publicX25519));      // F6EDB2996FF843651EA33630E497F0273CC625284A6ABA66FF2E3AAF98089D65

KeyPair keyPair = PublicKeyBox.GenerateKeyPair(secretX25519);
Console.WriteLine(Convert.ToHexString(keyPair.PublicKey)); // F6EDB2996FF843651EA33630E497F0273CC625284A6ABA66FF2E3AAF98089D65

// Or alternativley via multiplication with the base point
byte[] publicX25519Calc = ScalarMult.Base(secretX25519);
Console.WriteLine(Convert.ToHexString(publicX25519Calc));  // F6EDB2996FF843651EA33630E497F0273CC625284A6ABA66FF2E3AAF98089D65

The public key derived from the converted private key (with GenerateKeyPair() or by multiplication) is identical to the converted public key of OpenSSL.

---

With the converted keys, of course, public-key authenticated encryption works, e.g. with the NaCl.net library you are using (for simplicity, the same key pair is used on both sides):

using NaCl;

byte[] secretX25519 = Convert.FromHexString("C01A8724CB3C3EB03C1B015C1B91DEF56136391088EB5D1E60CB3AB56068D364");
byte[] publicX25519 = Convert.FromHexString("F6EDB2996FF843651EA33630E497F0273CC625284A6ABA66FF2E3AAF98089D65");

Curve25519XSalsa20Poly1305 crypter = new Curve25519XSalsa20Poly1305(secretX25519, publicX25519);

byte[] nonce = Encoding.UTF8.GetBytes("012345678901234567890123"); // static, for testing purposes only
byte[] msg = Encoding.UTF8.GetBytes("The quick brown fox jumps over the lazy dog");

byte[] ciphertext = new byte[msg.Length + 16];
crypter.Encrypt(ciphertext, msg, nonce);
Console.WriteLine(Convert.ToHexString(ciphertext)); // BBCA733F22D8BE302E1EB95E619FB954892C030F9A38CD7ABD1EAB6832A2379CADBFEF43A07788045037D1777CA9A2E17E63ECFC283EA68D6090A6

byte[] decrypted = new byte[ciphertext.Length - 16];
crypter.TryDecrypt(decrypted, ciphertext, nonce);
Console.WriteLine(Encoding.UTF8.GetString(decrypted)); // The quick brown fox jumps over the lazy dog