In two of our many PHP pods (we're running under Kubernetes) we have a problem with a Unix socket leak. Our PHP CLI process seems to be opening and not closing many unix sockets, until in an unrelated location PHP crashes with PHP Warning: stream_select(): You MUST recompile PHP with a larger value of FD_SETSIZE.
Other pods that we have also have some UNIX sockets open, but in small numbers so it's not a problem.
The perplexing thing is that we're not using UNIX sockets for anything. All communication with other processes is purely through TCP/IP. All the sockets are:
ESTABLISHED statess -x says that there have been no data either sent or received on these socketsnetstat says that they all have a RefCount of 3, if it means anythingss -xp also shows, that both ends of each socket are held by the same (one and only) PHP process.So... for some reason the PHP program is creating UNIX sockets, holding them open, holding on to both ends of that socket (two different file descriptors), and not letting them go.
In addition, there are no calls to stream_socket_pair or socket_create_pair in the code.
What I gather from this is that these sockets are created as a side-effect of some other PHP built-in function that I'm using. CURL is the primary suspect here, but it's also used plenty in other pods without issues. In fact I can't think of anything that these two pods are doing which others aren't.
What built-in PHP functions (including functions in extensions that come bundled by default with PHP) could result in an unintentional creation of a UNIX socket pair?
Added:
ss -xp looks like:
# ss -xp
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
u_str ESTAB 0 0 * 481304571 * 481304572 users:(("php",pid=7,fd=494))
u_str ESTAB 0 0 * 480934363 * 480934362 users:(("php",pid=7,fd=289))
u_str ESTAB 0 0 * 480899260 * 480899259 users:(("php",pid=7,fd=255))
u_str ESTAB 0 0 * 480853228 * 480853229 users:(("php",pid=7,fd=207))
u_str ESTAB 0 0 * 478659551 * 478659550 users:(("php",pid=7,fd=82))
u_str ESTAB 0 0 * 481263776 * 481263775 users:(("php",pid=7,fd=484))
u_str ESTAB 0 0 * 481014545 * 481014546 users:(("php",pid=7,fd=339))
u_str ESTAB 0 0 * 480847844 * 480847845 users:(("php",pid=7,fd=221))
u_str ESTAB 0 0 * 479446282 * 479446283 users:(("php",pid=7,fd=185))
u_str ESTAB 0 0 * 479430966 * 479430965 users:(("php",pid=7,fd=162))
u_str ESTAB 0 0 * 481391710 * 481391711 users:(("php",pid=7,fd=515))
u_str ESTAB 0 0 * 481013994 * 481013995 users:(("php",pid=7,fd=352))
u_str ESTAB 0 0 * 480884372 * 480884373 users:(("php",pid=7,fd=233))
.... many, may more lines ...
I found an answer myself. Posting here to help whoever next comes along. The culprit is... *drumroll*... CURL. In particular, curl_multi_init() in PHP 8.2 opens a pair of UNIX sockets, presumably for some sort of cross-thread synchronization.
Here's a piece of code that reproduces it:
$handles = [];
for ( $i = 0; $i < 10; $i++ )
$handles[] = curl_multi_init();
echo "Sleeping...";
sleep(10); // Here we have 20 unix sockets open
foreach ($handles as $h)
curl_multi_close($h); // According to PHP docs, this doesn't do anything in PHP 8, but it's good style anyway.
$handles = [];
echo "Sleeping again...";
sleep(10); // Here we have 2 unix sockets open... because the $h variable still holds the last handle
echo "Goodbye!";
I do not know why this behavior isn't present on my local computer. Perhaps it has something to do with the specific CURL or PHP version. In particular:
I have PHP 8.2.13 with CURL 8.5.0 (this doesn't have a problem) while the Kubernetes pod has PHP 8.2.16 with CURL 7.88.1 (this has a problem).
If someone knows why this doesn't happen on my machine, please let me know!
OK, now just to hunt down my own CurlMultiHandle leak - that's a much more manageable task and outside the scope of this question.