linuxrocksdbld-preloadtcmallocjemalloc

How to make sure which memory allocator will be used by RocksDB?


According to https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html it is sufficient to do the following to use a non-standard ie. more efficient memory allocator on Linux, in this case primarily for RocksDB:

# example: install jemalloc (on Debian)
apt install -y libjemalloc-dev

# set LD_PRELOAD before you start your Kafka Streams application
export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so"

Since with my workload I measured the same 4.5GiB memory consumption as without jemalloc I tried a run with tcmalloc too, which also showed the same memory usage. So I tried to analyse the usage of the dynamically linked libraries as follow.

The output of ldd /tmp/librocksdbjni16525886107314830425.so is:

    linux-vdso.so.1 (0x00007fffc3ba6000)
    /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 (0x00007f5e50dff000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5e51d94000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5e51d8f000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5e51d8a000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5e50bd3000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5e51ca1000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5e51c81000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5e509aa000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5e51d9f000)
    libunwind.so.8 => /lib/x86_64-linux-gnu/libunwind.so.8 (0x00007f5e51c66000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5e51c3b000)

The output of objdump -T /tmp/librocksdbjni16525886107314830425.so | grep malloc is:

0000000000000000      DF *UND*  0000000000000000 (GLIBC_2.2.5) malloc_usable_size
0000000000000000      DF *UND*  0000000000000000 (GLIBC_2.2.5) malloc
0000000000000000  w   D  *UND*  0000000000000000              mallocx
0000000000000000  w   D  *UND*  0000000000000000              malloc_stats_print
0000000000558360  w   DF .text  00000000000000e2  Base        _ZN7rocksdb23JemallocNodumpAllocatorD1Ev
00000000002fedd0 g    DF .text  0000000000000093  Base        rocksdb_jemalloc_nodump_allocator_create
0000000000558360  w   DF .text  00000000000000e2  Base        _ZN7rocksdb23JemallocNodumpAllocatorD2Ev
0000000000bf85e0  w   DO .data.rel.ro   00000000000000c0  Base        _ZTVN7rocksdb23JemallocNodumpAllocatorE
0000000000557fa0 g    DF .text  00000000000002f7  Base        _ZN7rocksdb26NewJemallocNodumpAllocatorERNS_24JemallocAllocatorOptionsEPSt10shared_ptrINS_15MemoryAllocatorEE
0000000000557bd0 g    DF .text  000000000000015b  Base        _ZN7rocksdb23JemallocNodumpAllocatorC1ERNS_24JemallocAllocatorOptionsE
0000000000557d30 g    DF .text  0000000000000096  Base        _ZN7rocksdb23JemallocNodumpAllocator11IsSupportedEPSs
00000000005582d0  w   DF .text  0000000000000008  Base        _ZNK7rocksdb23JemallocNodumpAllocator4NameEv
0000000000557bd0 g    DF .text  000000000000015b  Base        _ZN7rocksdb23JemallocNodumpAllocatorC2ERNS_24JemallocAllocatorOptionsE
00000000007eb510 g    DF .text  0000000000000005  Base        _Z12toku_xmallocm
0000000000301fb0 g    DF .text  000000000000000b  Base        rocksdb_options_set_dump_malloc_stats
0000000000557dd0 g    DF .text  00000000000001c3  Base        _ZN7rocksdb23JemallocNodumpAllocator14PrepareOptionsERKNS_13ConfigOptionsE
00000000007ebe30 g    DF .text  00000000000000e5  Base        _ZN8memarena17malloc_from_arenaEm
0000000000558450  w   DF .text  00000000000000bb  Base        _ZN7rocksdb23JemallocNodumpAllocatorD0Ev

Despite/due to this output I'm still unsure whether RocksDB uses the desired memory allocator.


Solution

  • According to https://github.com/jemalloc/jemalloc/issues/388 the following two commands do it:

    # negative test
    MALLOC_CONF=abort_conf:true,invalid_option:foo /layers/paketo-buildpacks_bellsoft-liberica/jre/bin/java -version
    
    MALLOC_CONF=stats_print:true /layers/paketo-buildpacks_bellsoft-liberica/jre/bin/java -version
    

    Additionally with the following command can be verified whether the dynamic link has happened:

    pmap 1 | grep malloc