Mutual authentication with Tomcat 7

I'm trying to set up a Java web service running in Tomcat 7 to use mutual (2-way) authentication. It seems like no matter what I do, connecting to the service on the secure port isn't working.

Here's what I did to create certificates and keystores and such:

//create the key and certificate for the tomcat server.
keytool -genkey -v -alias tomcat -keyalg RSA -validity 3650 -keystore tomcat.keystore

//create the key and certificate for the client machine.
keytool -genkey -v -alias clientkey -keyalg RSA -storetype PKCS12 -keystore client.p12

//export the client key
keytool -export -alias clientkey -keystore client.p12 -storetype PKCS12 -rfc -file client.cer

//import the client key into the server keystore
keytool -import -v -file client.cer -keystore tomcat.keystore

Here's the connector in the server.xml file:

<Connector port="8443"
    maxThreads="150"
    scheme="https"
    secure="true"
    sslProtocol="TLS"
    clientAuth="true"
    keystoreFile="tomcat.keystore"
    keystorePass="tomcat"
    truststoreFile="tomcat.keystore"
    truststorePass="tomcat"/>

The tomcat-users.xml file looks like this:

<tomcat-users>
    <role rolename="tomcat"/>
    <role rolename="admin"/>
    <!-- note that the actual values for CN, OU, O, L, ST are different, but they match the values created in the client certificate -->
    <user username="CN=name, OU=unit, O=org, L=locality, ST=state, C=US" password="null" roles="admin" />
</tomcat-users>

The following are set on startup:

-Djavax.net.ssl.keyStoreType=jks
-Djavax.net.ssl.keyStore=tomcat.keystore
-Djavax.net.ssl.keyStorePassword=tomcat
-Djavax.net.ssl.trustStore=tomcat.keystore
-Djavax.net.ssl.trustStorePassword=tomcat
-Djavax.net.debug=SSL

Finally, I copied the client.p12 file to my client machine, and imported it into Firefox's client certificates.

First problem: When I hit an endpoint on my service (example - https://my.server.com:8443/test) from Firefox, I get the response "Secure Connection Failed". SSL received a record that exceeded the maximum permissible length. (Error code: ssl_error_rx_record_too_long)

Second problem: I don't really want to run this connector on port 8443. I want to run it on port 7800 (which is our company standard for HTTPS). When I change the port on the Connector to 7800 and try to hit the endpoint (example - https://my.server.com:7800/test) then it never resolves the page.

So, somewhere I'm obviously missing a crucial piece. Can anyone see my error?

UPDATE: after feedback from @Dave G

Running the command:

openssl s_client -connect localhost:8443 -showcerts

produces the following output:

CONNECTED(00000003)
140642290976584:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:766:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 263 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

I also added -Djavax.net.debug=SSL to the startup. This generates the following in the beginning of the catalina.out file:

trustStore is: tomcat.keystore
trustStore type is : jks
trustStore provider is :
init truststore
adding as trusted cert:
  Subject: CN=localhost, OU=unit, O=org, L=Springfield, ST=MO, C=US
  Issuer:  CN=localhost, OU=unit, O=org, L=Springfield, ST=MO, C=US
  Algorithm: RSA; Serial number: 0x5485b5a5
  Valid from Mon Dec 08 14:28:53 UTC 2014 until Thu Dec 05 14:28:53 UTC 2024

adding as trusted cert:
  Subject: CN=William Jackson, OU=unit, O=org, L=Springfield, ST=MO, C=US
  Issuer:  CN=William Jackson, OU=unit, O=org, L=Springfield, ST=MO, C=US
  Algorithm: RSA; Serial number: 0x5485b6af
  Valid from Mon Dec 08 14:33:19 UTC 2014 until Sun Mar 08 14:33:19 UTC 2015

trigger seeding of SecureRandom
done seeding SecureRandom

And then a LOT of:

Ignoring unavailable cipher suite: <suite name>
Ignoring unsupported cipher suite: <suite name>

Solution

Ok - after digging a lot more, I finally got this working. Much thanks to @Dave G and this tutorial: Configuring two-way SSL authentication on Tomcat from which most of these instructions are paraphrased.

Generally, the steps to get mutual authentication functional are as follows:

Create a certificate for the tomcat server. The client has to trust this certificate.
Create a keystore for the tomcat server, and import the server certificate into it.
Create a certificate for the client. The server has to trust this certificate.
Import the client certificate into the server keystore
Update the tomcat server.xml file with the correct Connector XML.

The above steps are necessary on the server. Once completed, to set up the client, do the following:

Copy the client certificate from the server to the client.
Use the client certificate when communicating with the server (this process varies with the nature of the client application).

For the certificate configuration, I executed the following on the server machine:

# For the following commands, set the values in parenthesis to be whatever makes sense for your environment.  The parenthesis are not necessary for the command.

# This is an all-in-one command that generates a certificate for the server and places it in a keystore file, while setting both the certifcate password and the keystore password.
# The net result is a file called "tomcat.keystore". 

keytool -genkeypair -alias (serveralias) -keyalg RSA -dname "CN=(server-fqdn),OU=(organizationalunit),O=(organization),L=(locality),ST=(state),C=(country)" -keystore tomcat.keystore -keypass (password) -storepass (password)

# This is the all-in-one command that generates the certificate for the client and places it in a keystore file, while setting both the certificate password and the keystore password.
# The net result is a file called "client.keystore"

keytool -genkeypair -alias (clientalias) -keyalg RSA -dname "CN=(client),OU=(organizationalunit),O=(organization),L=(locality),ST=(state),C=(country)" -keypass (password) -keystore client.keystore -storepass (password) 

# This command exports the client certificate.  
# The net result is a file called "client.cer" in your home directory.

keytool -exportcert -rfc -alias (clientalias) -file client.cer -keypass (password) -keystore client.keystore -storepass (password)

# This command imports the client certificate into the "tomcat.keystore" file.

keytool -importcert -alias (clientalias) -file client.cer -keystore tomcat.keystore -storepass (password) -noprompt

Certificates should now be set up appropriately. The next step is to configure your connector in the tomcat server.xml. Add a connector element that looks like this:

<Connector port="8443"
    maxThreads="150"
    scheme="https"
    secure="true"
    SSLEnabled="true"
    truststoreFile="/full/path/to/tomcat.keystore"
    truststorePass="(password)"
    keystoreFile="/full/path/to/tomcat.keystore"
    keystorePass="(password)"
    clientAuth="true"
    keyAlias="serverkey"
    sslProtocol="TLS"/>

Note that in the above XML:

The "port" attribute can be whatever you want.
The "keystoreFile" and "truststoreFile" attributes should be full paths. Tomcat does not look in the same directory as server.xml by default.
The "keystorePass" and "truststorePass" attributes should match the (password) value you used in the creation of the tomcat.keystore file.
The "clientAuth" attribute must be set to "true". This is what triggers mutual authentication.

Additionally, in the server.xml, ensure that you DO NOT have an AprLifecycleListner defined. The XML for that listener will look something like this:

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

That element should be delete/commented out. The AprLifecycleListener does not get configured the same way as described above, and will not work with these instructions.

Restart tomcat. The server configuration should be complete.

I tested my work using Firefox, because it's easy to add client certificates to it. Open up Firefox and try to connect to an endpoint of your tomcat service on the port defined in your connector.

Ex: https://mytomcatdomain.com:8443/test

When you do this, you should get the standard alert from Firefox about an untrusted connection because we created a self-signed certificate for our Tomcat server. Add an exception for the certificate so that our client (Firefox) trusts our server (Tomcat).

Once you've added the exception, you should get a "Secure Connection Failed" message. The error code is "ssl_error_bad_cert_alert". This confirms that our Tomcat server is requesting authentication from the client. The request is failing because we have not configured Firefox to send our trusted client certificate yet.

To configure Firefox, we need to do a little more magic:

// Create a file called DumpPrivateKey.java.  The contents should look like so:
public class DumpPrivateKey {
public static void main(String[] args) throws Exception {
  final String keystoreName = args[0];
    final String keystorePassword = args[1];
    final String alias = args[2];
    java.security.KeyStore ks = java.security.KeyStore.getInstance("jks");
    ks.load(new java.io.FileInputStream(keystoreName), keystorePassword.toCharArray());
    System.out.println("-----BEGIN PRIVATE KEY-----");
    System.out.println(new sun.misc.BASE64Encoder().encode(ks.getKey(alias, keystorePassword.toCharArray()).getEncoded()));
    System.out.println("-----END PRIVATE KEY-----");
  }
}

Compile the java file with the following command:

javac DumpPrivateKey.java

Now we're going to use this little utility to extract a key from the client.keystore file we create above. Copy the client.keystore and client.cer files into the same directory as your DumpPrivateKey class. Execute the following:

# This extracts the client key from the client keystore

java DumpPrivateKey client.keystore (password) clientkey > clientkey.pkcs8

# This creates a client.p12 file that can be used by Firefox

openssl pkcs12 -export -in client.cer -inkey clientkey.pkcs8 -password pass:(password) -out client.p12

Note that in the above code, (password) should be the password you used to create the client.keystore.

Open up Firefox preferences. Click on the "Certificates" tab. Click on the "View Certificates" button. Click on the "Your Certificates" tab.

Click on the "Import" button and browse to the "client.p12" file that was created previously. You should be prompted to enter the password for the client certificate.

Assuming the "client.p12" was imported successfully, you can now refresh you Firefox page, and you should get a successful response from your Tomcat server endpoint.