I've spent the last few days searching google, blogs and MSDN looking for any small scrap of info on how "interop" or "mixed mode" debugging is implemented in Visual Studio.
I'm attempting to implement my own debugger for a custom VM (actually, it gets JIT-ed to native code, which means I'll have to rebuild callstacks for the normal native debugger whenever it enter's the JIT-ed code), but I can't find any info on how you can actually interact with VS's native debugger and do the fancy "native/managed" like transitions that VS's mixed mode debugging pulls off.
So far I've found a few things that provide useful tidbits, but not enough to actually interact with the debugger. The best, most useful articles I've found are:
How can I get started with writing a debugger that will work cooperatively with VS's native debugger?
Thanks!
I'm afraid there's not much documentation on this subject. The resources you mention are all pretty old at this point. My suggestion is to integrate with Visual Studio's newer debug engine (Concord). Concord is used as the debug engine for native debugging from Visual Studio 2012 forward and for all debugging in Visual Studio 2013 forward. One of the design goals of Concord was to simplify mixed mode debugging. It is also designed to be easily extensible.
I created some Concord extensibility documentation that may be of help. It is mainly focused on expression evaluators, but it has some good information for getting started with Concord. Another resource that may be of use is the source code for the Concord-based debugger in Python Tools for Visual Studio. This is a full implementation of a mixed mode debugger that integrates with Concord and allows mixed mode debugging with Python.