c++jemalloc

jeprof got stuck when handling dumped heap files

I want to profile memory allocation of my program. So I compile with JEMALLOC_PROF=1 and JEMALLOC_PROF_LIBUNWIND=1, and run with export MALLOC_CONF="prof:true,prof_active:true,lg_prof_interval:25".

The program did dump some heap files. However, I find I can't use jeprof to analysis this file. If I run with the following command, jeprof got stuck with the following error messages.

./jeprof --svg bin/tiflash/tiflash jeprof.910766.14.i14.heap > t.svg
Using local file bin/tiflash/tiflash.
Using local file jeprof.910766.14.i14.heap.
addr2line: DWARF error: could not find variable specification at offset 6cbe
addr2line: DWARF error: could not find variable specification at offset 3cec
addr2line: DWARF error: could not find variable specification at offset 1b26
addr2line: DWARF error: could not find variable specification at offset 372c
addr2line: DWARF error: could not find variable specification at offset 6fd6
addr2line: DWARF error: invalid or unhandled FORM value: 0x25

After some search, I find DWARF error: could not find variable specification at offset 6cbe means some symbol is missing, so jeprof can't handle. However, the program doesn't relate to the stuck.

From https://github.com/jemalloc/jemalloc/issues/2417, I know the DWARF error: invalid or unhandled FORM value: 0x25 is related to some DWARF issue. No furthermore hints though.

I want to know how to solve this problem, and let jeprof output an svg.

==== EDIT ====

After some further investigation, I find what jeprof did is call command

addr2line -f -C -e xxx -i </tmp/jeprof986545.sym

And the addr2line took lots of time, and output an error

addr2line -f -e xxx 000000000aa3b336


addr2line: DWARF error: invalid or unhandled FORM value: 0x25
_ZN5gimli4read5dwarf14Dwarf$LT$R$GT$4unit17h2c6ef33ecf31a970E
backtrace.e024eb3ce795f700-cgu.3:?

==== EDIT ====

I have switched to the latest binutils 2.42, and still addr2line stucks forever.

Solution

  • I have figured the problem out by myself.

    1. addr2line: DWARF error: invalid or unhandled FORM value: 0x25 This is because my binutils is too old. I compiled the latest binutils, and it could handle 0x25
    2. addr2line stucks because the binary is in debug mode, and is too big(around 2.5GiB) When I build the binary in release mode. addr2line can output the result in less than a minute.