debugging gstreamer file-descriptor lsof gstreamer-1.0

How to debug gstreamer pipeline with leaking file descriptors after gst_object_unref()?

I have a custom pipeline that looks roughly like this in gstreamer shorthand:

gst-launch-1.0 rtspsrc location=rtsp://<url-for-stream> ! rtph264depay ! h264parse ! imxvpudec ! *any-sink*

any-sink doesn't matter, could be fakesink, imxipusink, or whatever (I'm on imx6 platform using freescale imx plugins). I can output to whichever sink I want and the issue is the same.

This type of pipeline works fine in gst-launch-1.0 because it doesn't need to clean itself up properly, but I need to use it inside my C++ application using direct GST API. This means I use myPipeline = gst_pipeline_new("custom-pipeline"), then allocate each plugin by name, link them, and run the pipeline. I later have a requirement to stop the pipeline and call gst_object_unref(myPipeline). When doing this, I observe file descriptors being left behind. I later need to start the pipeline all over again, and so the leak is compounding. This needs to happen often enough that the leaking descriptors give me an exception:

GLib-ERROR **: Creating pipes for GWakeup: Too many open files

I can profile the open files with lsof...

lsof +E -aUc myGstApplication        
lsof: netlink UNIX socket msg peer info error
COMMAND    PID USER   FD   TYPE     DEVICE SIZE/OFF     NODE NAME
myGstApplication 5943 root    3u  unix 0xabfb6c00      0t0 11200335 type=STREAM
myGstApplication 5943 root   11u  unix 0xd9d47180      0t0 11207020 type=STREAM

... many more, depending on how long it runs...

myGstApplication 5943 root   50u  unix 0xabe99080      0t0 11211987 type=STREAM

I appear to get two new 'type=STREAM' file descriptors each time i unref() and rebuild the pipeline.

This is all fine and dandy to see the descriptors in lsof, but I don't know how to track down where these files are coming from in the code. Does any of the lsof output actually lead me to better debug information, for instance? . How do I track down where these leaks are really coming from and stop them? There has be be a better way... right?

I suspect rtspsrc gstreamer pipeline element is having something to do with this, but rtspsrc is itself a morass of underlying gstreamer implementation (udpsrcs, demuxers, etc, etc.) I'm not convinced that it's a bug within rtspsrc, because so many other people appear to use this one without reproducing the same thing. Could I be doing something in my application code that can bring about this behavior in a non-obvious way?

Any help is much appreciated, thanks!

Solution

Well researched & interesting question!

According to the lsof output, the leaking file descriptors seem to originate from socketpair syscalls. You confirm this with strace:

strace -fe socketpair myGstApplication

After this, you could leave out filtering for the socketpair sycall, and look through the full strace output, trying to understand what these FDs are used for. I tried this with gst-launch-1.0, with inconclusive results. These FDs seem to be set readonly on both ends and nothing is ever transferred... so they must be used for control/coordination between several threads/subprocesses of the same application only.

Next try would be gdb:

gdb -ex 'break socketpair' -ex run myGstApplication

When it halts at the breakpoint, look at the stacktrace with the bt command. Probably installing the debug packages of gstreamer is a good idea to get more readable stacktraces.

HTH :)