[SOLVED] Why isn't IIS cleaning up the old worker processes (w3wp.exe) on pool recycle leading to website out of memory exception?

Why isn't IIS cleaning up the old worker processes (w3wp.exe) on pool recycle leading to website out of memory exception?

I have an asp.net-mvc site and recently I am getting an out of memory exceptions on my web server. I only have 1 application pool and we recent set IIS to recycle after it hits a certain limit. I went in the other day and saw 4 w3wp.exe processes running (each with ~1.8GB memory being used)

I assume that during the recycle process, it's not killing the old worker process and eventually I get out of memory exceptions on my website because the box only has 8GB memory. I can add memory to the box but I am concerned why these old processes are not being cleaned up.

Are there any recommendations to figure out why this recycle process is not killing the old w3wp.exe processes and leaving them running? Any suggestions around understand both root cause or even workarounds to avoid this risk going forward?

Solution

I had similar issue when I was running things like FFMpeg.exe or some PDF conversion with WPF graphics, IIS process won't shutdown and would issue memory not found errors. The problem is not with IIS, but some deadlocks in the process which blocks even after crashing.

Workaround is, divide your website in to two separate websites, one should only do transaction processing with database which usually does not have crashes. The logic like video/photo conversion, PDF conversion, or any other logic that may cause crash should be moved to other web service. And use HTTP call from your website internally to process them over web services.

Now, in this case, there is still no way to get around web service process crashing, so I decided to recycle application pool worker every 100 requests (I chose this number after watching few requests, on an average it would go beyond 1GB only after hitting 200 requests) and I turned application pool into Web Garden by making 4 process per pool.

Advantage of this setup is, you can move your web service process to other machines easily in future, you can increase/decrease number of processes per pool. And your main website, which is simply doing transaction process becomes highly responsive as it is not affected by process recycles of web services.