[SOLVED] Measure java service performance during machine startup?

Measure java service performance during machine startup?

We have several java applications which run as Windows services using Tankui's java service wrapper. Our customer is encountering a scalability problem when the machine is starting up (first 30 seconds).

Traditionally, we've used jconsole/visualvm to monitor Java Virtual Machine health. But these tools aren't very good for capturing java.exe performance during a machine startup.

I've googled for an answer and the best search result I found was a paper titled "Measuring the Startup Time of a Java Virtual Machine" but they resorted to instrumenting an app with a JNI call http://www.mii.lt/olympiads_in_informatics/pdf/INFOL073.pdf. We would prefer a less invasive approach.

What is a good external tool or technique capable of capturing java.exe stats (thread counts, heap use, etc.) during machine startup?

@djmorton asked to define the scalability problem. The problem is that when the application is monitoring 100 system objects everything is fine. But when the data increases to 1,000 system objects then

1,100 JVM threads run concurrently
Thread X is holding a write lock on a Java read-write lock C1 is busy updating a part of a system object using Hibernate/c3p0/JDBC/MySQL
Thread Y is holding structure P's JVM object intrinsic lock and is waiting to obtain a read lock on Java read-write lock C1 to read another part of the system object from the database using Hibernate/c3p0/JDBC/MySQL
Thread Z is waiting to obtain structure P's JVM object intrinsic lock.

Solution

We ended up using JProfiler in offline-mode. JProfiler lets you set up triggers based on JVM startup. JProfiler let us grab various JVM telemetry data like thread counts, plus Java lock contention, thread dumps, etc. during Java service startup.

We had to work around one Windows file permissions issue. JProfiler wants to create a temp file but in this environment, the JProfiler is running in a Windows service using the LocalSystem account. The work around is to temporarily set the service up to run as Administrator.