We have one .NET webapi application (with a service autostart provider).
The application is built by our buildserver, and the output is deployed to multiple servers and requests to the application is routed through a loadbalancer.
The site has been performing well for several months. A couple of days ago we noticed that the applicationpool was using 3x more memory than it was the other day, on two out of 3 servers.
The servers that now uses more memory than the others is doing so even if we iisreset, reboot the server etc. The applicationpool is allocating 3x more memory than the other immediately after it starts.
And now to the super-weird stuff....
To find out what was being held up in the memory, I "attached" a dotTrace profiler (remote, using remoteagent.exe) to one of the servers. The pool seemed to recycle and once it was booted it did no longer allocate 3x memory (so the memory snapshot was useless). I "deattached" the profiler from the worker process and noticed that the application pool was no longer using 3x memory. I rebooted the server, and It was still not using 3x memory.
So... It seems like just attaching the profiler to the w3wp process fixed my problem, in a persisten way, since it's still working with the "low memory usage" after iisreset and reboots.
Tried the same procedure on server #2 which suffered the same problem and it also fixed my problem.
It seems like dotTrace does something magic to my workerprocesses :) ?
Can anyone figure out the cause of this? I have more servers (in other environments) with the same problem, but I would like to know the root cause of this problem.
Short answer: I've found the cause to be Microsoft Monitoring Agent's profiler which hooks on to my w3wp processes.
Long answer:
Details in the eventlog
.NET Runtime version 4.0.30319.34014 - The profiler was loaded successfully. Profiler CLSID: '{AD5651A8-B5C8-46CA-A11B-E82AEC2B8E78}'. Process ID (decimal): xxxx. Message ID: [0x2507].
led me to the conclusion that another profiler was attaching to the w3wp's. Searching for the CLSID in the registry made it obvious that the Monitoring agent was involved. The profiler can be disabled by first browsing the windows registry to:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W3SVC
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WAS
and changing:
Environment = COR_PROFILER={AD5651A8-B5C8-46ca-A11B-E82AEC2B8E78} Cor_Enable_Profiling=1
to:
Environment = COR_PROFILER={AD5651A8-B5C8-46ca-A11B-E82AEC2B8E78} Cor_Enable_Profiling=0
I guess that there's a better way to disabling profiling than changing the registry directly, (maybe uninstalling Microsoft Monitoring Agent?)
...why the profiler causes 3x memory consumption is still a mystery.