What exactly is a custom MTA for the Sitecore ECM? Is it nothing more than a mail server? Does anybody know what the requirements are, or specific instructions on how to set one up?
Yes, it's just a mail server. You have to have a license that supports the AnyMTA feature in order to use it. After that, it's just configuration for which you can find instructions in the documentation. You just have to set the UseLocalMTA setting to true and then set the SMTP settings.