Whilst trying to implement Passkey login using AWS Cognito, I found a guide written by AWS (https://github.com/aws-samples/webauthn-with-amazon-cognito/). It shows how to use lambda triggers to respond to challenges and to implement the flow. My solution uses React on the UI and ASP Net on the API and, though the demo code is a nodejs implementation using the library Fido2-Lib, I found I could implement almost the same in C# using the Fido2NetLib library.
Everything was going fine, all moving parts generally aligned around creating and parsing credentials, even the AWS lambdas for Cognito seemed to be playing nicely, until the crunch point came with logging in. I started to get error:1E08010C:DECODER routines::unsupported when the public key and signature were verified. I looked into this thinking first that the key was in an unexpected format. I realised that the Fido2-Lib in the demo code outputs a PEM format key, whereas the Fido2NetLib doesn't provide that option and instead provides COSE format CBOR object (inline with the standard - tho, as I write this I assume the PEM key from Fido2-Lib is also a CBOR inside the PEM format).
I double checked the contents of the key and ensured they were all pretty standard and acceptable by NodeJS Crypto
1: 2 - key type Elliptic Curve format
3: -7 - algorithm used ES256
-1: 1 - curve type P-256
-2: buffer(32) - x co-ordinate
-3: buffer(32) - y co-ordinate
using that information, I wrote a test program to just test the verify method, and tried adding PEM format (the '---- BEGIN PRIVATE KEY ----') no joy, I tried, taking the contents of the CBOR map in the private key and (having learned about DER format) create a DER format key, which I believe crypto can accept, using the following sample code, but no joy, the same UNSUPPORTED ROUTINE error.
const coseKey = Buffer.from(publicKeyCredJSON.publicKey, 'base64');
// Parse COSE key
const parsedKey = cbor.decodeFirstSync(coseKey);
// Extract necessary information
const keyType = parsedKey.get(1); // Key type
const algorithm = parsedKey.get(3); // Algorithm
const xCoord = parsedKey.get(-2);
const yCoord = parsedKey.get(-3);// Key value
// Convert key data to buffer
const publicKey = Buffer.concat([Buffer.from([0x04]), xCoord, yCoord]);
// Sample data and signature to verify (replace with your actual data and signature)
const rawAuthnrData = Buffer.from(challengeAnswerJSON.response.authenticatorData, 'base64');
const rawClientData = Buffer.from(challengeAnswerJSON.response.clientDataJSON, 'base64');
// Create verifier
const verifier = crypto.createVerify('SHA256');
// Provide data
verifier.update(rawAuthnrData);
verifier.update(rawClientData);
// Verify signature
const isValid = verifier.verify(publicKey, challengeAnswerJSON.response.signature, 'base64');`
I appreciate this is a nebulous topic, and having tried modifying the verify call to crypto to pass more parameters, understand the format of the key from Fido2NetLib and how it differs to Fido2-Lib I thought I'd reach out on here to see if I'm bumping into something silly, others have already solved.
It is true nodejs crypto accepts the PEM and DER formats defined by OpenSSL (which it uses internally); technically these are not the only possible PEM and DER formats, but in practice they are or until recently were the de-facto standards for cryptography-related data. Traditionally it defaulted to PEM and you needed to explicitly specify format=DER and in some cases type, but OpenSSL 3's new API (which your version is apparently using) may have changed this.
However, I don't know where you 'learned' about DER, but your posted code does not create anything even remotely similar to DER. It does convert the EC point to an X9-and-SEC1 format, which is the most common standard and is used in OpenSSL's and thus nodejs' PEM and DER formats, but it is not by itself either PEM or DER or even a valid key.
If you have nodejs 16 up, which I expect you do if you have OpenSSL 3, it also supports JWK format as used in JOSE, and this is (not accidentally) much closer to COSE. Try {format:'jwk', key:{perRFC7518}} where the minmal RFC7518 spec for EC public key is
{kty:'EC',crv:'P-256',x:bufferx.encode('base64url'),y:buffery.encode('base64url')}
For older nodejs you'll need in principle to construct the DER format defined in RFC5480 (or its PEM equivalent, but that's more work and no benefit); this includes an outer header, an AlgorithmIdentifier containing id-ecPublicKey and the curve OIDs, and a BIT STRING containing the point in SEC1 format (same as X9 with the exclusion of hybrid, which is not an issue for you). However as the point length is fixed per curve, for a given curve this DER structure is actually a fixed prefix plus the point, and you can copy the fixed prefix from a throwaway key (also for P-256!) you create with either nodejs or OpenSSL (they are the same thing here).