crashautomationlldb

Scripting LLDB to obtain a stack trace after a crash


I'm trying to add the capability of generating a stack trace from a core dump on mac automatically when one of our tests crashes.

I was able to do it pretty easily on linux with

gdb --batch --quiet -ex "thread apply all bt" -ex "quit" <binary> <core file> 2> /dev/null

But I'm having some trouble doing the same on mac (OSX 10.8) with lldb. To start off, the version of lldb I'm using is lldb-310.2.37.

My initial approach was to use the -s option and pass in a script file like so:

target create -c <core file> <binary>
thread backtrace all
quit

Initially I had some trouble which I think was caused by missing a newline at the end of the script file which caused lldb to not exit, but after that was fixed, I'm getting the following: Executing commands in 'lldbSource'.

(lldb)  target create -c <core file> <binary>
Core file '<core file>' (x86_64) was loaded.
(lldb)  thread backtrace all
error: Aborting reading of commands after command #1: 'thread backtrace all' failed with error: invalid thread
Aborting after_file command execution, command file: 'lldbSource' failed.

The funny thing is, after that happens, we're still running lldb, and issuing 'thread backtrace all' manually works just fine.

So, approach #2 was to create a python script and use their python API (I tried this before figuring out the initial blocker I described was due to a missing newline).

My script:

import lldb
debugger = lldb.SBDebugger.Create()
target = debugger.CreateTarget('<binary>')
if target:
 process = target.LoadCore('<core file>')
 if process:
  print process.exit_description
  for thread in process:
   print 'Thread %s:' % str(thread.GetThreadID())
   print '\n'.join(str(frame) for frame in thread)

The problem I'm having with this approach is that process.exit_description is returning None (and so is every other thing I've tried; LLDB's python API documentation is almost completely useless).

The output I'm looking for from that call is something similar to the following:

Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff8aca4670 libsystem_c.dylib`strlen + 16, stop reason = signal SIGSTOP
    frame #0: 0x00007fff8aca4670 libsystem_c.dylib`strlen + 16
libsystem_c.dylib`strlen + 16:
-> 0x7fff8aca4670:  pcmpeqb (%rdi), %xmm0
   0x7fff8aca4674:  andl   $0xf, %ecx
   0x7fff8aca4677:  shll   %cl, %eax
   0x7fff8aca4679:  pmovmskb %xmm0, %ecx

This is output automatically by LLDB proper when loading a core file. I don't necessarily need the assembly dump, but I want at least the thread, frame and reason.

I think the first approach I used, if it could be made to work, would be ideal, but either way is OK for me. I don't have control over the LLDB version that's going to be used unfortunately, so I can't just update to latest and see if it's a bug that was fixed.

Other approaches to get the desired output are also welcome. For context, this is going to be called from a perl script.


Solution

  • TOT lldb from lldb.llvm.org has a new "--batch" mode to work pretty much like the gdb batch mode, and fixes some bugs that made command-line source commands behave better. If you can build your own lldb, you can get these fixes now, otherwise you'll have to wait till the next Xcode update.

    The exit_description is None because your process didn't exit, it crashed. Note that at least on OS X several threads can have had simultaneous exceptions by the time the process gets around to crashing, it isn't really useful to say the process crashed. You have to ask the threads. The stop status that lldb prints out when a thread stops is available with the GetStatus method.

    stream = lldb.SBStream()
    process.threads[0].GetStatus(stream)
    print stream.GetData()
    

    That doesn't seem to have a very useful help string.