I am trying to use the -mk option of procdump (version 11) to get the kernel part of the stack trace of a 64 bit Windows process. The kernel dump is created, but opening it in WinDbg, all I get is a single stack frame (DbgkpLkmdSnapThreadInContext) rather than a full stack trace:
0: kd> k20
# Child-SP RetAddr Call Site
00 fffff50e`7b54df30 00000000`00000000 nt!DbgkpLkmdSnapThreadInContext+0x9d
This was done on Windows 11 22H2 (22621.4169), but I also got the same result on Windows 10.
Am I doing anything wrong? Or is the -mk option of procdump broken?
More details: As an example, I call procdump from Windows Terminal (run with administrator rights) to create a dump of that very same Windows Terminal process (pid 46304):
PS C:\dev\Tools> .\procdump.exe -mk -ma 46304 "C:\dev\test\dump.dmp"
ProcDump v11.0 - Sysinternals process dump utility
Copyright (C) 2009-2022 Mark Russinovich and Andrew Richards
Sysinternals - www.sysinternals.com
[08:23:43] Dump 1 initiated: C:\dev\test\dump.dmp
[08:23:43] Dump 1 writing: Estimated dump file size is 535 MB.
[08:23:44] Dump 1 complete: 536 MB written in 1.5 seconds
[08:23:44] Dump 1 kernel: C:\dev\test\dump.Kernel.dmp
[08:23:44] Dump count reached.
The user mode dump (dump.dmp) shows that the process is inside a syscall:
0:000> k9
# Child-SP RetAddr Call Site
00 00000048`c24ff558 00007ffd`199b53fa win32u!NtUserGetMessage+0x14
01 00000048`c24ff560 00007ff6`f0b840b5 user32!GetMessageW+0x2a
02 (Inline Function) --------`-------- WindowsTerminal!WindowEmperor::WaitForWindows+0x61 [C:\__w\1\s\src\cascadia\WindowsTerminal\WindowEmperor.cpp @ 125]
03 00000048`c24ff5c0 00007ff6`f0b8e2a6 WindowsTerminal!WindowEmperor::HandleCommandlineArgs+0x535 [C:\__w\1\s\src\cascadia\WindowsTerminal\WindowEmperor.cpp @ 102]
04 00000048`c24ff8a0 00007ff6`f0b927f2 WindowsTerminal!wWinMain+0x166 [C:\__w\1\s\src\cascadia\WindowsTerminal\main.cpp @ 118]
05 (Inline Function) --------`-------- WindowsTerminal!invoke_main+0x21 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 118]
06 00000048`c24ff980 00007ffd`19e7257d WindowsTerminal!__scrt_common_main_seh+0x106 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
07 00000048`c24ff9c0 00007ffd`1bc2af28 kernel32!BaseThreadInitThunk+0x1d
08 00000048`c24ff9f0 00000000`00000000 ntdll!RtlUserThreadStart+0x28
where the RIP points to 00007ffd`198d1534:
win32u!NtUserGetMessage:
00007ffd`198d1520 4c8bd1 mov r10, rcx
00007ffd`198d1523 b804100000 mov eax, 1004h
00007ffd`198d1528 f604250803fe7f01 test byte ptr [7FFE0308h], 1
00007ffd`198d1530 7503 jne win32u!NtUserGetMessage+0x15 (7ffd198d1535)
00007ffd`198d1532 0f05 syscall
00007ffd`198d1534 c3 ret
So I do expect to get some kernel frames.
But opening the kernel dump (dump.Kernel.dmp) created by procdump, all I see is the DbgkpLkmdSnapThreadInContext stack frame, even after switching explicitly to my process. On the other hand, when collecting a live kernel dump, I do see that the stack continues indeed:
0: kd> k20
# Child-SP RetAddr Call Site
00 fffff50e`7bb86ba0 fffff803`48e5ffa5 nt!KiSwapContext+0x76
01 fffff50e`7bb86ce0 fffff803`48e613c7 nt!KiSwapThread+0xaa5
02 fffff50e`7bb86e30 fffff803`48f40176 nt!KiCommitThreadWait+0x137
03 fffff50e`7bb86ee0 fffff803`48f51c93 nt!KeWaitForSingleObject+0x256
04 fffff50e`7bb87280 ffff8968`68f4d670 nt!KeWaitForMultipleObjects+0x5d3
05 fffff50e`7bb874e0 ffff8968`68f4d2af win32kfull!xxxRealSleepThread+0x310
06 fffff50e`7bb87600 ffff8968`68f508ab win32kfull!xxxSleepThread2+0xaf
07 fffff50e`7bb87650 ffff8968`68f4dadc win32kfull!xxxRealInternalGetMessage+0x15bb
08 fffff50e`7bb87950 ffff8968`687d6ed2 win32kfull!NtUserGetMessage+0x8c
09 fffff50e`7bb879e0 fffff803`4902b605 win32k!NtUserGetMessage+0x16
0a fffff50e`7bb87a20 00007ffd`198d1534 nt!KiSystemServiceCopyEnd+0x25
0b 00000048`c30ff8e8 00007ffd`199b53fa win32u!NtUserGetMessage+0x14
0c 00000048`c30ff8f0 00000000`00000000 user32!GetMessageW+0x2a
But procdump seems to be unable to capture it. Why?
Context: We already use procdump in our test system to automatically create dumps of hanging/crashed processes. I try to extend it to also create a kernel dump (stack trace only, of course) of the process in question.
TL;DR: "Kernel mini dumps" aka "kernel triage dumps" are broken on current Windows versions (at least till Windows 11 22H2, most likely also later) if there are too many logical cores in the processor (any logical core count above or close to 20 is dangerous). Reason: The dumps are limited to 1MB, and contain information for every core. If there are too many cores, the 1MB limit is reached with the core information alone.
After quite some digging (via ProcMon and disassemblers), I think I know what is going on: First of all, procdump -mk is using the WriteKernelMinidumpCallback feature of the WinAPI MiniDumpWriteDump() to create the "kernel mini dump".
That API is internally using the undocumented syscall NtSystemDebugControl(). Amongst other things, this syscall implements the "kernel mini dump", in that context called "kernel triage dump". One can find some information about that syscall online, e.g. here, here or here. The SYSDBG_COMMAND parameter (first parameter) of NtSystemDebugControl() needs to be set to SysDbgGetTriageDump = 29.
Moreover, one needs to pass in a buffer that gets filled with the data, and the size of the buffer. The maximum supported buffer size is 1MB. Attempting to pass in a larger size still produces a dump that is at most 1MB. This can be seen when disassembling NtSystemDebugControl() in ntoskrnl.exe (I used version 10.0.19041.5007, Windows 10 22H2), which contains code such as
usedBufferSize = 0x100000;
if (givenBufferSize <= 0x100000)
usedBufferSize = givenBufferSize;
where givenBufferSize is the size of the buffer that one specified in the call to NtSystemDebugControl(), and usedBufferSize is the size actually used for the dump.
Then the code calls the internal function DbgkCaptureLiveDump() (to create the kernel triage dump), which contains a call to DbgkpLkmdSnapGlobals(). That function basically gets the processor information (via KeQueryActiveProcessorCountEx(ALL_PROCESSOR_GROUPS), KeGetPrcb() and KeEnumerateProcessorDpcs()). In my case, I have 16 real cores with SMT, so 32 logical cores. Thus, KeQueryActiveProcessorCountEx(ALL_PROCESSOR_GROUPS) returns 32. The code loops over every "logical processor" (meaning every logical core) and gets its "kernel processor control block" (KPRCB). It then apparently writes each one to the triage dump buffer. The thing is, the KPRCB structure is massive, and grows from Windows version to version. On Windows 11 23H2, it seems to have a size of 0xBF00. In my case (Windows 10 22H2), I can see from the disassembly that it has a size of 0xAF00 bytes.
Given a 1MB buffer, we can fit 23 KPRCB into that buffer. So if there are more than 23 logical cores, the whole buffer is filled with just the processor information, leaving not space for the actual interesting data (call stacks etc.).
On later Windows versions, it is worse because the size of the KPRCB structure is even larger (Windows 11 23H2: 0xBF00, meaning that 21 cores already fill the dump).
Indeed, I used a virtual machine (Windows 10 22H2) and varied the number of cores available to the VM. With <=21 cores, WinDbg is able to display proper call stacks from the kernel triage dump. But with >=22 cores, WinDbg fails to do so. The threshold is very close to the 23 KPRCBs.
Also, using a dump on 32 cores, the WinDbg command !prcb 0x15 (0x15=21) shows information, while !prcb 0x16 (0x16=22) fails.
Hence, kernel triage dumps are simply broken (or at least useless) on modern CPUs with many cores due to the size restriction to 1MB.
In case anyone wants to play around with NtSystemDebugControl() directly: github.com/Sedeniono/livedump.