I am trying to find the lock contention in my code and I found the futexes.stp in the standard systemtap release. But I have a couple of questions with the output:
stap -d /usr/local/bin/cruncher ~/futexes.stp
(and then I run cruncher application in another term)
auditd[1803] lock 0x564dd507f228 contended 2 times, 119 avg us
auditd[1803] lock 0x564dd507f254 contended 28 times, 458040 avg us
rs:main Q:Reg[1837] lock 0x5563b67c2044 contended 7 times, 2352067 avg us
...
cruncher[17021] lock 0x7f6c9c57914c contended 2 times, 46 avg us
cruncher[17021] lock 0x7eff50 contended 82 times, 16 avg us
cruncher[17021] lock 0x7f6c40901f24 contended 20 times, 109 avg us
cruncher[17021] lock 0x7f6f38e0633c contended 4471 times, 2106 avg us
cruncher[17021] lock 0x7dae7c0 contended 8861 times, 26 avg us
cruncher[17021] lock 0x7f6cc2f14560 contended 2 times, 9 avg us
cruncher[17021] lock 0x7f6c8685d024 contended 5 times, 95 avg us
cruncher[17021] lock 0x7f6cd9e2ded4 contended 6 times, 428 avg us
cruncher[17021] lock 0x7f6c5196937c contended 44 times, 484 avg us
cruncher[17021] lock 0x7f6ba20c0290 contended 4 times, 8 avg us
...
So here are the questions:
What all those constructs with 0x7f.......... stack addresses? I am sure it's not from my application. Are those library or kernel mutexes (or condvars) on behalf of my application? How do I verify that?
Those "locks" with short addresses are mine. I can verify them with gdb's "info symbol " command, but what will be a good way from stap to automatically decode it into a symbol?
I am using Linux 4.6.2 mainline kernel built on the same box, with latest systemtap from their git repository. My cruncher application was built with -g and the symbols are not stripped.
Thanks a lot in advance!
The lock 0x7f... addresses are userspace pointers passed to the futex(2) syscalls. The script just stores them verbatim. They are likely to point into the interior of a glibc pthreads-mutex type object.
As to which object ... an address-to-symbol lookup stage would have to be added. It could probably work by intercepting pthread mutex initialization with another probe to assemble a table of pthread locks. Or by adding usymdata(...) lookups to the incoming futex addresses. See the mutex-contention.stp sample script for use of both techniques.