I have a Go program that runs continuously, and I've noticed that both the number of goroutines and open file descriptors increase steadily over time (daily). However, the number of active network connections does not grow proportionally.
When inspecting the goroutine dump, I found that most of the long-lived goroutines are: net/http.(*persistConn).writeLoop net/http.(*persistConn).readLoop
Additionally, many of the open file descriptors are similar to:
myprogram 8 root 26u sock 0,8 0t0 349484146 protocol: TCP
This leads me to suspect there might be a leak or misuse of http.Client, possibly due to connections being kept alive indefinitely.
However, I'm unsure how to proceed with confirming or debugging this. Could someone suggest:
How to confirm whether http.Client or http.Transport is leaking connections/goroutines?
Diagnostic tools or metrics that could help track down the leak?
Any ideas or tips would be greatly appreciated!
goroutine 1030476 [select, 3878 minutes]:
net/http.(*persistConn).writeLoop(0xc02af42240)
/usr/local/go/src/net/http/transport.go:2458 +0xf0
created by net/http.(*Transport).dialConn in goroutine 1030485
/usr/local/go/src/net/http/transport.go:1800 +0x1585
goroutine 667283 [select, 5476 minutes]:
net/http.(*persistConn).readLoop(0xc029285440)
/usr/local/go/src/net/http/transport.go:2261 +0xd3a
created by net/http.(*Transport).dialConn in goroutine 667247
/usr/local/go/src/net/http/transport.go:1799 +0x152f
lsof -p [pid]
myprogram 8 root 26u sock 0,8 0t0 349484146 protocol: TCP
myprogram 8 root 27u sock 0,8 0t0 350094405 protocol: TCP
myprogram 8 root 29u sock 0,8 0t0 350354169 protocol: TCP
-----------------update--------------------
About whether the body is closed, i found that in my http client, i add a Roundtrip which logging request and response. The code is like:
func (l *RoundTripper) Log(r *http.Response) {
if r == nil {
return
}
respDump, err := httputil.DumpResponse(r, true)
if err != nil {
l.Logger.Error(fmt.Sprintf("DumpResponse error:%v", err))
}
l.Logger.Info(fmt.Sprintf("Response: %s\n", respDump))
}
And it will call httputil.DumpResponse, in this function i think it will read and close body every time.
Because there are many calls of http.Client, i just couldn't find which code cause the leak. Any ideas or tips for finding out the caller would be greatly appreciated!
As i can find out from your information , likely facing a connection leak due to improper usage of the http.Client or its underlying http.Transport , and the steadily increasing number of goroutines means the connections are not closed correctly or reuse (because of ReadLoop/WriteLoop ) .
Here are what you can do about it :
A-to confirm http.client or http.transport Leak :
1-try reuse the client and dont create a new one for each request
2-use defer resp.body.Close() (if u did not)
3- you can use http transport metrics/debug logging (like this:GODEBUG=http2debug=2 ./myprogram)
B-u can also prevent creating many http.transport , just use a single transport instance and reuse it , like this one :
transport := &http.Transport{
MaxIdleConns: 10,
MaxIdleConnsPerHost: 10,
IdleConnTimeout: 30 * time.Second,
}
hope this be helpful