W3WP.EXE using 100% CPU - where to start?

An ASP.NET web app running on IIS6 periodically shoots the CPU up to 100%. It's the W3WP that's responsible for nearly all CPU usage during these episodes. The CPU stays pinned at 100% anywhere from a few minutes to over an hour.

This is on a staging server and the site is only getting very light traffic from testers at this point.

We've running ANTS profiler on the server, but it's been unenlightening.

Where can we start finding out what's causing these episodes and what code is keeping the CPU busy during all that time?

1. Standard Windows performance counters (look for other correlated activity, such as many GET requests, excessive network or disk I/O, etc); you can read them from code as well as from perfmon (to trigger data collection if CPU use exceeds a threshold, for example)
2. Custom performance counters (particularly to time for off-box requests and other calls where execution time is uncertain)
3. Load testing, using tools such as Visual Studio Team Test or WCAT
4. If you can test on or upgrade to IIS 7, you can configure Failed Request Tracing to generate a trace if requests take more a certain amount of time
5. Use logparser to see which requests arrived at the time of the CPU spike
6. Code reviews / walk-throughs (in particular, look for loops that may not terminate properly, such as if an error happens, as well as locks and potential threading issues, such as the use of statics)
7. CPU and memory profiling (can be difficult on a production system)
8. Process Explorer
9. Windows Resource Monitor
10. Detailed error logging
11. Custom trace logging, including execution time details (perhaps conditional, based on the CPU-use perf counter)
12. Are the errors happening when the AppPool recycles? If so, it could be a clue.