I am about to setup SmarterMail v9.0 on our Windows 2008 server (IIS7) and would first like to know what some security considerations are when opening up port 25 and/or 587 - ie how to prevent relaying, etc.
Thank you.
You must not accept email from untrusted users/sources which is not bound for domains you control.
An open relay is a mail server which allows anyone on the Internet to email anyone else, without verifying that either the source or the destination is known - thus, a relay.
You can check that the source is known by looking for a trusted IP subnet, or by requiring authentication before mail can be sent (via LOGIN over TLS, GSSAPI [called "Integrated Windows Authentication" or whatever], X.509 client certs, or the like).
You can check that the destination is known by comparing it to the list of domains for which your mail server will be the "last stop" (or a relay to another domain you control).
Either a known source or a known destination should be sufficient, but you may also want to make sure that mail inbound for your domains is at least borderline valid (originates from a domain with an MX server, for instance).
Separately, you must be conscious of DoS issues (rate limit inbound mail), and the ability to use your server to send backscatter spam. Backscatter is when I connect to your mail server and say, "why yes, I am unsuspecting_target@not_my_domain.com, please queue up this message for not_an_address@yourdomain.com". Then your mail server delivers a "bounce" message to the unsuspecting target. To mitigate this, you can verify that the recipient is known before accepting mail, or limit the rate at which mail can be accepted from one host, or try to check that the host delivering a message is authorized to use that envelope sender.
These are all well-solved problems.