I'm trying to set up a web server in Delphi XE3 using Indy and OpenSSL that can serve traffic over both HTTP and HTTPS connections on the same port.
I have seen two main approaches to this, and neither seems to work for me.
First: Up-front TLS/SSL. This involves reading the first few bytes of the stream to look for the "Client-Hello" part of the non-secure handshake and (if found) invoking the server SSL handshake response, but if I do that, the OpenSSL library does not recognize the handshake because I've stripped off the leading bytes of the message.
Second: TLS after STARTTLS (or equivalent). This involves sending a special set of characters (STARTTLS) which will be immediately followed by "Client-Hello". The server then leaves the entire SSL handshake message intact to pass to the OpenSSL library. The problem with this approach is that most web browsers don't support it (RFC 2817).
For a summary of the two approaches, look here: What happens on the wire when a TLS / LDAP or TLS / HTTP connection is set up?)
How can I support SSL and non-SSL traffic on the same port using TIdHTTPServer and OpenSSL?
What you ask for is not possible with Indy out of the box. Indy's default SSL implementation uses OpenSSL's traditional API, where it does all of its own socket I/O, so it needs direct access to the complete handshake data without letting you peek at the data first. However, all is not lost. You have a couple of choices:
1) use libpcap/Winpcap to capture and look at the first few bytes of raw data coming off the wire for new connections before the socket provides it to your application.
2) write your own TIdIOHandler derived class that uses OpenSSL's newer BIO API, or Microsoft's SChannel API, so you are in control of the socket I/O and can read incoming bytes yourself and look at them before pushing them into the encryption engine for processing.
UPDATE: there is actually a 3rd option, using Indy's default classes:
3) in the TIdHTTPServer.OnQuerySSLPort event, set the VUseSSL parameter to False for all requests, thus turning off TIdHTTPServer's automatic handling of the SSL/TLS handshake. Then, in the TIdHTTPServer.OnConnect event (which is fired after the OnQuerySSLPort event), you can peek the first few bytes directly from the socket connection (using the OS's socket API recv() function with the AContext.Binding.Handle socket handle and the MSG_PEEK flag). If the peeked bytes represent the beginning of a "Client-Hello" SSL/TLS message, then type-cast the AContext.Connection.IOHandler object to TIdSSLIOHandlerSocketBase and set its PassThrough property to False to initiate OpenSSL's normal SSL/TLS handshake processing. Since the peeked bytes will still be in the socket's internal buffer, OpenSSL will be able to read them.