Preventing debugging session from pausing after each inferior exits

I'm debugging a tree of processes using gdb's very handy multiple-inferior support:

(gdb) set detach-on-fork off
(gdb) set schedule-multiple on
(gdb) set follow-fork-mode parent
(gdb) break PostgresMain
(gdb) break PostmasterMain

and now need to let things run until I hit one of the future breakpoints in some yet to be spawned inferior.

However, gdb seems to be "helpfully" pausing whenever an inferior exits normally, or at least blocking cleanup of the inferior so that its parent's wait() can return:

(gdb) c
[New process 16505]
process 16505 is executing new program: /home/craig/pg/bdr/bin/pg_config
Reading symbols from /home/craig/pg/bdr/bin/pg_config...done.
[Inferior 2 (process 16505) exited normally]
(gdb) info inferior
  Num  Description       Executable        
* 2    <null>            /home/craig/pg/bdr/bin/pg_config 
  1    process 16501     /usr/bin/make     
(gdb) inferior 1
[Switching to inferior 1 [process 16501] (/usr/bin/make)]
[Switching to thread 1 (process 16501)] 
#0  0x0000003bc68bc502 in __libc_wait (stat_loc=0x7fffffffbc78) at ../sysdeps/unix/sysv/linux/wait.c:30
30          return INLINE_SYSCALL (wait4, 4, WAIT_ANY, stat_loc, 0,
(gdb)

so I have to endlessly:

(gdb) inferior 1
(gdb) c

to carry on. About 70 times, before I hit the desired breakpoint in a child of a child of a child.

I think what's happening is that gdb treats process exit as a stop event, and since non-stop is set to off (the default) it stops all threads in all inferiors when one thread stops. However, this inferior has terminated, it isn't a normal stop event, so you can't just cont it, you have to switch to another process first.

Is there some way to stop gdb pausing at each inferior exit? I would've expected follow-fork-mode parent with schedule-multiple on to do the trick, but gdb seems to still want to stop when an inferior exits.

I guess I'm looking for something like a "skip proc-exit", or a virtual signal I can change the handler policy on so it doesn't stop.

set non-stop on seems like it should be the right answer, but I suspect it's broken for multiple inferiors.

If I use non-stop on, then after the first exit trap, gdb's internal state indicates that inferior 1 is running:

(gdb) info inferior
  Num  Description       Executable        
* 1    process 20540     /usr/bin/make     
(gdb) info thread
  Id   Target Id         Frame 
* 1    process 20540 "make" (running)
(gdb) cont
Continuing.
Cannot execute this command while the selected thread is running.

but the kernel sees it as blocked on ptrace_stop:

$ ps -o "cmd,wchan" -p 20540
CMD                         WCHAN
/usr/bin/make check         ptrace_stop

... and it makes no progress until gdb is detached, or it's killed. Signals to the process are ignored, and interrupt in gdb has no effect.

I'm using GNU gdb (GDB) Fedora 7.7.1-18.fc20 on x86_64.

Solution

Improvement of this answer.

Apparently gdb.execute() inside a handler causes a segmentation fault. So I modify it to use post_event() instead.

Add the following to .gdbinit

python
def switch_inferior_and_continue(x):
    gdb.execute("inferior %d" % x)
    gdb.execute("continue")

def exit_handler(event):
    has_threads = [ inferior.num for inferior in gdb.inferiors() if inferior.threads() ]
    if has_threads:
        has_threads.sort()
        gdb.post_event(lambda: switch_inferior_and_continue(has_threads[0]))

gdb.events.exited.connect(exit_handler)
end