Is it possible to add MathJax functionality to the Sandcastle documentation tool?
If principally yes, where the reference to Mathjax should be added?
Alternatively, is there any other possibility to include mathematical notation into Sandcastle documentation?
AFAIK - no.
There's a build component that inserts additional resources into the topics such as stylesheets and scripts. The easiest approach would be to create a plug-in that runs just before the BuildAssembler step that modifies the BuildAssembler configuration file (sandcastle.config in the working folder) by inserting an extra script element into the Additional Header Resources Component configurations. There are multiple instances, one for each output format in each section for conceptual and API topics. It would also need to copy the required script to the output folder.
Quoted from MathML in Topics.