[SOLVED] Certificate Verification Error Using Express Gateway

I'm having an issue getting Express Gateway to connect to a backend service when using certificates generated with OpenSSL. Whenever the gateway tries to connect to the service I get this error in the log:

project_edge_express_1    | 2019-12-03T23:44:32.189Z [EG:gateway] debug: request matched condition in proxy policy
project_edge_express_1    | 2019-12-03T23:44:32.189Z [EG:policy] debug: proxying to https://api.project.local/, POST /oauth/token
project_edge_express_1    | 2019-12-03T23:44:32.266Z [EG:policy] warn: unable to verify the first certificate

I had assumed that this was a certificate issue and went through several rounds of regenerating certs. Below is the output from my latest iteration:

C:\..\carbon> openssl x509 -in .\ca\intermediate\certs\api.project.local.cert.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=Oklahoma, CN=Development Intermediate CA
        Validity
            Not Before: Dec  3 04:09:36 2019 GMT
            Not After : Nov 30 04:09:36 2029 GMT
        Subject: C=US, ST=Oklahoma, L=Oklahoma City, CN=api.project.local
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b6:f6:d1:60:88:db:8e:a4:b3:bd:8e:61:02:8b:
                    ...
                    69:6b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Server
            Netscape Comment:
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier:
                88:72:74:76:DC:3A:3B:AD:47:7D:85:60:F4:35:2C:76:E3:F2:0D:E0
            X509v3 Authority Key Identifier:
                keyid:9B:95:50:21:22:4A:66:48:5A:43:E2:0D:8D:A4:25:93:8E:4D:4F:27
                DirName:/C=US/ST=Oklahoma/L=Oklahoma City/CN=Development Root CA
                serial:10:00

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:api.project.local
    Signature Algorithm: sha256WithRSAEncryption
         b6:2e:ee:69:a6:4d:0e:4d:dc:f4:42:31:4e:77:10:56:fa:3d:
         ...
         c6:eb:40:17:08:82:4c:0f

When I log into the container where express is running, though, and perform a manual certificate check OpenSSL reports the cert as valid:

C:\..\project> docker exec -it project_edge_express_1 /bin/bash
root@efc66f266d19:/usr/src/app# openssl s_client -connect api.project.local:443
CONNECTED(00000003)
depth=2 C = US, ST = Oklahoma, L = Oklahoma City, CN = Development Root CA
verify return:1
depth=1 C = US, ST = Oklahoma, L = Oklahoma City, CN = Intermediate CA
verify return:1
depth=0 C = US, ST = Oklahoma, L = Oklahoma City, CN = api.project.local
verify return:1
---
Certificate chain
 0 s:/C=US/ST=Oklahoma/L=Oklahoma City/CN=api.project.local
   i:/C=US/ST=Oklahoma/L=Oklahoma City/CN=Development Intermediate CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIF3DCCA8SgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwbTELMAkGA1UEBhMCVVMx
...
kBOe8nsEvMnG60AXCIJMDw==
-----END CERTIFICATE-----
subject=/C=US/ST=Oklahoma/L=Oklahoma City/CN=api.project.local
issuer=/C=US/ST=Oklahoma/L=Oklahoma City/CN=Development Intermediate CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2129 bytes and written 269 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: CC...B8
    Session-ID-ctx:
    Master-Key: 0A...D631C75477            PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - ...

    Start Time: 1575416738
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes

I even tried manually disabling certificate validation in express gateway. Here is my current config:

http:
  port: 8080
admin:
  port: 9876
  host: localhost
apiEndpoints:
  api:
    - host: localhost
      paths: '/ip'

    - host: localhost
      paths: '/oauth/token'
      methods: ['POST']

    - host: localhost
      paths: '/v1/*'
serviceEndpoints:
  httpbin:
    url: 'https://httpbin.org'
  project:
    url: 'https://api.project.local'
policies:
  - cors
  - log
  - proxy
pipelines:
  default:
    apiEndpoints:
      - api
    policies:
      - proxy:
          secure: false
          condition: 
            name: pathExact
            path: /oauth/token
          action:
            serviceEndpoint: oauth

I'm not sure where to go from here except to rely on the kindness of strangers.

UPDATE 3 Dec 2019 8:05 PM CST:

I've written a small POC in node using the https lib and run the same request using that lib. The request went through as expected. I've also validated the request using cURL which was successful as well.

UPDATE 4 Dec 2019 6:31 AM CST:

Here's the Dockerfile which builds the container running Express Gateway in case it's of some use:

FROM node:10

COPY ./ca/intermediate/certs/ca-chain.cert.pem /usr/local/share/ca-certificates/ca-chain.crt
RUN chmod 644 /usr/local/share/ca-certificates/* && update-ca-certificates

WORKDIR /usr/src/app

COPY edge/gateway/package*.json ./

RUN npm install

COPY edge/gateway .

EXPOSE 8080
CMD [ "node", "server.js" ]