Application: ASP.NET, C#
Framework: 4.6.2
Authentication mode:- Windows Authentication-Allow.
Platform:64-bit
Server: Win-2012
IIS:8.5
Is being migrated to Azure platform. In testing phase - automated test tool, sends rapid requests. Those requests could be anything like, dummy requests, malicious requests, page doesn't exists, page with inserted javascript or existing page with wrong parameters. etc. Basically tool will try to break/hack your application. Somehow as soon as this tool start testing, our app-pool is getting crashed. Sometimes it runs for a while but doesn't go long. During testing when we monitored IIS worker process crashes frequently.
Where as manual browser testing works fine.
I have tried following things:
1. Verified CPU and memory usage, they are normal.
2. Verified http logs: Those requests which are captured shows correct error codes. E.g. Page which doesn't exits shows 500, lot of request shows unauthorized 401.1, some correct request will also shows 200.
3. Application is set for Windows Authentication-Allow mode. Automated test tool uses some valid credentials. Though we see lot of requests are unauthorized.
4. Tried with different machines and different user credentials, every time IIS crashes.
5. Issue is reproducible at both DEV and UAT environments.
6. Was able to pull crash dumps but not able to read/debug it as of now. doing research on how to read them.
7. Application is writing logs into db. But that is not costly operation too. It may not be the root cause. Since sometimes with first or second request also it will start failing.
8. Compared with application those pass these kind of testing, nothing special in our configuration.
9. Tried changing requests per seconds. It didn't help.
10. Verified application event logs: it mainly shows following 2 errors.
A) Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: ntdll.dll, version: 6.3.9600.19153, time stamp: 0x5b93ffa7
Exception code: 0xc00000fd
Fault offset: 0x0000000000030461
Faulting process id: 0x4d48
Faulting application path: C:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\windows\SYSTEM32\ntdll.dll
B) Same error with clr.dll.
So not able to find the exact reason why the worker process is keep getting crashed. IIS will allow 5 recycling in a row and then stops doing it then the application will remain in unavailable state (503 error). Appreciate any suggestions/hints to reach to the root cause.
So there were two things.
1. Why the app pool was crashing?
Looking at the event log it was clean that there was overflow.
2. However I didn't know where it was. For that I look into crash dump. I got the location. Then I fixed it. Thanks @Wiz, it was into logging code.