How to find out why Nginx return 400 while use it as http/2 load balancer?

I'm implementing a http/2 proxy myself, and I'm using Nginx as load balancer.

When I use Nginx as h2c load balancer, it works:

server {
    listen 8443 http2;

    location / {
        error_log /Users/jiajun/nginx_error_log.log debug;
        grpc_pass grpc://127.0.0.1:2017;
    }
}

Run it:

$ go run example/grpc_client/main.go
calling to 127.0.0.1:2019
2019/01/28 11:50:46 gonna call c.SayHello...
2019/01/28 11:50:46 Greeting: Hello world

And Nginx access log is:

127.0.0.1 - - [28/Jan/2019:11:50:46 +0800] "POST /helloworld.Greeter/SayHello HTTP/2.0" 200 18 "-" "grpc-go/1.16.0" "-"

---

But it does not work while I'm serving it under SSL:

server {
    listen 8443 ssl http2;
    ssl_certificate     /Users/jiajun/127.0.0.1.crt;
    ssl_certificate_key /Users/jiajun/127.0.0.1.key;

    location / {
        error_log /Users/jiajun/nginx_error_log.log debug;
        grpc_pass grpc://127.0.0.1:2017;
    }
}

Run it:

$ go run example/grpc_client/main.go
calling to 127.0.0.1:2019
2019/01/28 11:53:06 gonna call c.SayHello...
2019/01/28 11:53:06 could not greet: rpc error: code = Unavailable desc = transport is closing
exit status 1

And, Nginx access log is:

127.0.0.1 - - [28/Jan/2019:11:53:06 +0800] "PRI * HTTP/2.0" 400 157 "-" "-" "-"

---

So I'm trying to find out why it return a 400 while I'm using SSL. But Nginx does not record the process in error log even I've set log level to debug(and it had compiled with --with-debug option).

Is there any good idea to debug it?

Looks like Nginx doesn’t think it’s talking HTTP/2 as the go client is sending the connection preface message ("PRI * HTTP/2.0") which Nginx thinks is a real message.

The likely issue is that Nginx has not stated in the SSL/TLS handshake that it supports HTTP/2 via ALPN (or the older NPN). Which version of OpenSSL (or equivalent) was Nginx built with? Run nginx -V to see this. Ideally you want 1.0.2 or above to support ALPN.

Also check that your SSL/TLS cipher is not using one of the ciphers forbidden for HTTP/2. Config like below (generated by the Mozilla SSL config generator) should ensure the right ciphers are preferred:

# modern configuration. tweak to your needs.
ssl_protocols TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_prefer_server_ciphers on;