I thought they were the same thing except I've just found out they are not!
Using IE8 I can get the same webpage to display in 3 different ways:
Standards mode. The XHTML Strict DOCTYPE is at the top of the webpage.
Same as above except that 'compatibilty mode' is clicked in IE8. Now the webpage is displayed slightly weird.
If I remove the DOCTYPE from the webpage (this forces quirks mode?). Now the webpage is displayed very badly and is not the same as when viewed in 'compatibility mode'. At this point clicking 'compatibilty mode' makes no difference.
Quirks mode is basically an IE5 compatibility mode. It is triggered by not having a valid <!DOCTYPE> declaration. The main effect is that it causes the browser to use the IE5 box-model, which means that all your paddings, margins and borders, and anything else which affects the size of a box will be incorrect.
Compatibility mode is an IE7-compatibility mode (IE9 also has an IE8-compatibility mode) which is triggered either by a meta tag specifying that the page should use it, or by the user's browser configuration.
Both can also be specified explicitly in the developer tools window.
Compatiblity mode is occasionally useful for testing (since IE7 doesn't have the developer tools window) and for intranet sites where the company is too cheap to update their code to cope with the newer browser. (but having said that, compatibility mode is not an exact replica of IE7; it has it's own bugs and issues, so for most testing, you're better off sticking with a real copy of IE7)
Quirks mode should never be used. Who in their right mind would want to be compatible with IE5?