.netperformancecpu

Diagnosing runaway CPU in a .NET production application


Is there a tool that can help me figure out why we are seeing runaway CPU in a managed application?

What I am not looking for:

  1. Process Explorer, it has this awesome feature that lets you see CPU per thread, but you do not get managed stack traces. Also, it requires a fairly proficient user.

  2. WinDbg + SOS, it could probably be used to figure out what is going on, by grabbing a bunch of dumps. But it is non-trivial to automate and a bit to heavy for this.

  3. Fully fledged profiler (like dotTrace or Redgate), licensing is complex and the tool is an overkill which requires a reasonably heavy install.

What I am looking for:

  1. A simple EXE file (without any installer) I can send to a customer. After they run it for 10 minutes, it generates a file that they send to me. The file contains details on the threads that consumed the most CPU and their stack traces during that time.

Technically, I know that a tool like this can be created (using ICorDebug), but I do not want to invest any time if such a tool already exists.

So, is there anything like this?


Solution

  • The basic solution

    1. Grab managed stack traces of each managed thread.
    2. Grab basic thread statistics for each managed thread (user mode and kernel time)
    3. Wait a bit
    4. Repeat (1-3)
    5. Analyze the results and find the threads consuming the largest amount of cpu usage, present the stack traces of those threads to the user.

    Managed Vs. Unmanged Stack Traces

    There is a big difference between managed and unmanged stack traces. Managed stack traces contain information about actual .Net calls whereas unmanaged ones contain a list of unmanaged function pointers. Since .Net is jitted the addressed of the unmanaged function pointers are of little use when diagnosing a problem with managed applications.

    managed stack not that useful

    How do you get an unmanaged stack trace for an arbitrary .Net process?

    There are two ways you could get managed stack traces for an managed application.

    What is better in production?

    The CLR Debugging APIs have a very important advantage over the profiling ones, they allow you to attach to a running process. This can be critical when diagnosing performance issues in production. Quite often runaway CPU pops up after days of application use due to some unexpected branch of code executing. At that point of time restarting the app (in order to profile it) is not an option.

    cpu-analyzer.exe

    So, I wrote a little tool that has no-installer and performs the basic solution above using ICorDebug. Its based off the mdbg source which is all merged into a single exe.

    It takes a configurable (default is 10) number of stack traces for all managed threads, at a configurable interval (default is 1000ms).

    Here is a sample output:

    C:\>cpu-analyzer.exe evilapp
    ------------------------------------
    4948
    Kernel Time: 0 User Time: 89856576
    EvilApp.Program.MisterEvil
    EvilApp.Program.b__0
    System.Threading.ExecutionContext.Run
    System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal
    System.Threading._ThreadPoolWaitCallback.PerformWaitCallback
    
    ... more data omitted ...
    

    Feel free to give the tool a shot. It can be downloaded from my blog.

    EDIT

    Here is a thread showing how I use cpu-analyzer to diagnose such an issue in a production app.