From browsing on this site and elsewhere, I've learned that serving websites as XHTML at present is considered harmful.
Delivering XHTML and serving it as application/xhtml+xml isn't supported by the majority of people browsing at present, delivering xhtml as text/html is at best a placebo for myself, and at worst a recipe for breaking sites usually when you least need it happening.
So we end up back at html 4.01. If I instead serve my pages as html 4.01, is it possible to use SVG or any other XML-based language on the page?
If so, how?
In HTML you won't be able to insert SVG directly.
You can embed SVG files with <object>/<embed> and in cutting-edge browsers (Opera, Safari) also <img> and CSS background-image.
You can put SVG in data: URI to avoid using external files.
Simple mathematical expressions can be written with help of Unicode and basic HTML/CSS (Opera 9.5 supports large chunk of MathML via CSS). For anything more complex you'll need to use images, like Wikipedia does.
HTML misinterprets namespace prefixes, so you won't be able to (properly) use other XML markup with HTML DOM. HTML5 has data-* attributes for application-specific markup additions. For metadata consider Microformats.
However if you want to embed XML only for non-browsers (robots), then you could use HTML-compatible XHTML subset and HTTP content negotiation to send proper XML with proper type to clients that understand it (if you thoroughly test page in both XML and HTML modes, then it won't be harmful).