pgx tls connection throws client cert invalid error for valid cert

I'm trying to use pgx to make a TLS connection to a postgres 10 db.

My connection string is similar to: "host='my-host.com' port='5432' dbname='my-db' user='my-db-user' sslmode='verify-full' sslcert='/path/to/db_user.crt' sslkey='/path/to/db_user.key' sslrootcert='/path/to/ca_roots.pem'"

When I run this directly with psql on the command-line, it works, so the cert and key files must be valid. db_user.crt and db_user.key are both PEM files. (the command-line also works with sslmode='verify-full', so the rootcert should also be ok)

But when I initialize a pgx pool with that connection string, it fails with:

FATAL: connection requires a valid client certificate (SQLSTATE 28000)

Is go expecting something other than PEM? Or is there a different way ssl cert and key pair is supposed to be initialized with pgx?

Code

import (
    "context"
    "fmt"
    "os"

    "github.com/jackc/pgx/v4"
    "github.com/jackc/pgx/v4/pgxpool"
)

type mockLogger struct{}

func (ml *mockLogger) Log(ctx context.Context, level pgx.LogLevel, msg string, data map[string]interface{}) {
    fmt.Printf("[%s] %s : %+v\n", level.String(), msg, data)
}

func connect() error {
    connStr := "host='my-host.com' port='5432' dbname='my-db' user='my-db-user' sslmode='verify-full' sslcert='/path/to/db_user.crt' sslkey='/path/to/db_user.key' sslrootcert='/path/to/ca_roots.pem'"
    poolCfg, err := pgxpool.ParseConfig(connStr)
    if err != nil {
        return err
    }
    poolCfg.ConnConfig.Logger = &mockLogger{}
    poolCfg.ConnConfig.LogLevel = pgx.LogLevelTrace
    fmt.Printf("using connection string: \"%s\"\n", poolCfg.ConnString())

    connPool, err := pgxpool.ConnectConfig(context.TODO(), poolCfg)
    if err != nil {
        return err
    }

    connPool.Close()
    return nil
}

func main() {
    if err := connect(); err != nil {
        fmt.Printf("%+v\n", err)
        os.Exit(1)
    }
}

Output from calling connect():

using connection string: "host='my-host.com' port='5432' dbname='my-db' user='my-db-user' sslmode='require' sslcert='/path/to/db_user.crt' sslkey='/path/to/db_user.key' sslrootcert='/path/to/ca_roots.pem'"
[info] Dialing PostgreSQL server : map[host:my-host.com]
[error] connect failed : map[err:failed to connect to `host=my-host.com user=my-db-user database=my-db`: server error (FATAL: connection requires a valid client certificate (SQLSTATE 28000))]
failed to connect to `host=my-host.com user=my-db-user database=my-db`: server error (FATAL: connection requires a valid client certificate (SQLSTATE 28000))

Solution

Summary

Turns out for go, the cert pointed to by sslcert needed to contain the full client cert chain.

When /path/to/db_user.crt contained the client cert followed by client cert chain, the pgx connection worked.

Whereas the psql command worked in both cases:

when sslcert was just the leaf client cert without the chain
when sslcert contained client cert + chain

Not sure why psql was fine without the full chain, but it works now.

Details

Under-the-hood, pgx uses the pgconn module to create the connection. That, in turn, is just calling tls.X509KeyPair on the contents of the sslcert and sslkey files.

pgconn/config.go:

func configTLS(settings map[string]string, thisHost string, parseConfigOptions ParseConfigOptions) ([]*tls.Config, error) {
    [...]
    sslcert := settings["sslcert"]
    sslkey := settings["sslkey"]
    [...]
    if sslcert != "" && sslkey != "" {
        [...]
        certfile, err := ioutil.ReadFile(sslcert)
        if err != nil {
            return nil, fmt.Errorf("unable to read cert: %w", err)
        }
        cert, err := tls.X509KeyPair(certfile, pemKey)
        if err != nil {
            return nil, fmt.Errorf("unable to load cert: %w", err)
        }
        tlsConfig.Certificates = []tls.Certificate{cert}