So I have a grpc service in Golang, and I want to expose an HTTP API on top of it, so I added grpc-gateway to the service.
When I try to make an HTTP request to my service it fails with a weird error 500 Internal Server Error caused by: stream error: stream ID 19; PROTOCOL_ERROR
This happens right after the load balancer forwards the request to the http port (the port on which grpc-gateway runs)
Its a flaky error, requests fail with it most of the times but sometimes requests pass without errors
looking at the request in the browser I see that when the request is sent from the frontend it adds this header TE: trailers and according to a comment on this thread https://github.com/golang/go/issues/29125#issuecomment-676831990 the TE header can cause this problem. So I tried to remove the TE header using traefik middleware but even that didn't solve the problem, even after removing the header before it hits the http port the request still fails with the same error.
then I turned on GODEBUG=http2debug=2 and found this in the logs
http2: Framer 0xc000272e00: read HEADERS flags=END_HEADERS stream=5 len=29
http2: decoded hpack field header field ":authority" = "dev-env.something.com"
http2: decoded hpack field header field ":method" = "POST"
http2: decoded hpack field header field ":path" = "/v1/someAPI/Create"
http2: decoded hpack field header field ":scheme" = "http"
http2: decoded hpack field header field "x-forwarded-host" = "dev-env.something.com"
http2: decoded hpack field header field "x-forwarded-port" = "443"
http2: decoded hpack field header field "sec-fetch-dest" = "empty"
http2: decoded hpack field header field "x-forwarded-for" = "XX.XXX.XXX.X"
http2: decoded hpack field header field "sec-fetch-mode" = "cors"
http2: decoded hpack field header field "accept" = "application/json, text/plain, */*"
http2: decoded hpack field header field "te" = "trailers"
http2: decoded hpack field header field "x-real-ip" = "XX.XXX.XXX.X"
http2: decoded hpack field header field "user-agent" = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"
http2: decoded hpack field header field "accept-encoding" = "gzip, deflate, br"
http2: decoded hpack field header field "sec-fetch-site" = "same-origin"
http2: decoded hpack field header field "referer" = "https://dev-env.something.com/"
http2: decoded hpack field header field "x-forwarded-proto" = "https"
http2: decoded hpack field header field "accept-language" = "en-US,en;q=0.5"
http2: decoded hpack field header field "x-forwarded-server" = "traefik-xxxxxxxxxxx"
http2: decoded hpack field header field "origin" = "https://dev-env.something.com"
http2: decoded hpack field header field "content-type" = "application/json"
http2: decoded hpack field header field "content-length" = "12"
http2: Framer 0xc000272e00: read DATA flags=END_STREAM stream=3 len=12 data="{\"name\":\"test\"}"
http2: Framer 0xc000272e00: wrote RST_STREAM stream=3 len=4 ErrCode=PROTOCOL_ERROR
http2: Framer 0xc000272e00: wrote WINDOW_UPDATE len=4 (conn) incr=12
http2: Framer 0xc000272e00: wrote PING len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
http2: Framer 0xc000272e00: read SETTINGS flags=ACK len=0
http2: Framer 0xc000272e00: read PING flags=ACK len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
Now even the logs aren't really helpful, either that or I dont know how to read them and I am missing something obvious
The curl request is of the form curl -X POST -H "Content-Type: application/json" "https://domain.name.com/v1/someAPI/Create" -d "{\"name\": \"test\" }"
Could someone tell me whats wrong, is this a bug in stdlib? If not it would still be a big help if someone could tell me how to read the http2 logs or where to look for the source of these logs in the standard library
So no bug in Standard library, it was a human error. A misconfiguration on the load balancer was directing the http request the grpc server half of the times and the http server the other times (both grpc server and the grpc-gateway reverse proxy were served on different ports)
Whenever the request hit the grpc server it accept the request because the grpc server runs on an H2C server but it wasn't able to translate it hence resulting in the error.