WLST connection to admin port always fails on first attempt

In our public testing environment the first attempt to connect in WLST always fails. The second attempt then always works and connection is established. I was unable to find anything similar using google so I hope someone here can help.

On the first attempt I get;

<Oct 7, 2014 1:49:03 PM EEST> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Oct 7, 2014 1:49:03 PM EEST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
Traceback (innermost last):
File "<console>", line 1, in ?
File "<iostream>", line 22, in connect
File "<iostream>", line 653, in raiseWLSTException
WLSTException: Error occured while performing connect : Error getting the initial context. There is no server running at t3s://xxx.xxx.xxx.xxx:nnnn

dumpStack() shows the following;

javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://xxx.xxx.xxx.xxx:nnnn: Destination unreachable; nested exception is:
    java.net.SocketException: Write failed: Broken pipe; No available router to destination]
javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://xxx.xxx.xxx.xxx:nnnn: Destination unreachable; nested exception is:
    java.net.SocketException: Write failed: Broken pipe; No available router to destination]

After this I try to connect again and this happens;

Connecting to t3s://xxx.xxx.xxx.xxx:nnnn with userid adminuserid ...
Successfully connected to Admin Server 'ourTestServer' that belongs to domain 'ourTestDomain'.

I've tried this about 10 times now and it always does the same, first attempt fails and second attempt works. Each time I've used IP and not a name for the address. This is in Linux (RHEL6) with WLS 10.3.6 with latest security patches.

I assume this line from the server's log is related as it happens 29 seconds after the info lines posted above;

####<Oct 7, 2014 1:49:32 PM EEST> <Warning> <Socket> <machinename.domain.local> <ourTestServer> <[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1412678972504> <BEA-000449> <Closing socket as no data read from it on xxx.xxx.xxx.xxx:nnnn during the configured idle timeout of 25 secs>

I'm at a loss as invariably the first attempt fails and second attempt works.

Solution

I had the same problem on WebLogic 11g running on RHEL 6.5. This little snippet before running wlst.sh worked around it for me (separate other properties with spaces if you need them):

export WLST_PROPERTIES="-Djava.security.egd=file:/dev/./urandom"

Or you can add the equivalent to your command line if you're not using the wlst.sh script to start the JVM.

I found the answer here: https://community.oracle.com/thread/1036828?start=0&tstart=0

It apparently comes down to an issue with the random number generator on "low entropy" machines. See Section 11.28.8. http://docs.oracle.com/cd/E14571_01/doc.1111/e14773/weblogic_server_issues.htm

My server machine in question is very lightly used at the moment, so it makes sense.