After upgrading to TIBCO BusinessWorks (BW) 5.10 (most likely not related) and TIBCO Runtime Agent (TRA) 5.7.4, JDBC connections using the Microsoft SQL Server JDBC driver (com.microsoft.sqlserver.jdbc.SQLServer - sqljdbc4.jar) times out. This happens for example in the TIBCO Designer when using "Test Connection" on a JDBC Connection.
Note that TRA 5.7.4 includes JRE 1.6.0_30.
Steps to resolve the problem:
Add the following line to your BW and Designer .tra files:
java.property.TIBCO_SECURITY_VENDOR=j2se
You may also have to perform the above for other TIBCO software such as Hawk and Administrator, depending on your setup.