How do I use the HotSpot DTrace probes on SmartOS?

On Mac OS X, I can find the HotSpot probes of running Java programs by running:

cody.mello@ashur ~ (1) % sudo dtrace -ln 'hotspot*:::'
Password:
Invalid connection: com.apple.coresymbolicationd
   ID   PROVIDER            MODULE                          FUNCTION NAME
165084  hotspot46      libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-clinit
165085  hotspot46      libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-concurrent
165086  hotspot46      libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-end
165087  hotspot46      libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-erroneous
165088  hotspot46      libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-error
165089  hotspot46      libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-recursive
...

But if I create a simple Java program and run it on SmartOS:

cody@101901c9-6d66-ea32-fe42-f1fbebd4bf99 ~ % cat Loop.java 
class Loop {

    public static void main(String[] args) throws InterruptedException {
        while (true) {
                Thread.sleep(5000);
        }
    }
}
cody@101901c9-6d66-ea32-fe42-f1fbebd4bf99 ~ % javac Loop.java 
cody@101901c9-6d66-ea32-fe42-f1fbebd4bf99 ~ % java Loop

I can't find any probes:

cody@101901c9-6d66-ea32-fe42-f1fbebd4bf99 ~ (255) % pfexec dtrace -ln 'hotspot*:::'
   ID   PROVIDER            MODULE                          FUNCTION NAME
dtrace: failed to match hotspot*:::: No probe matches description

Is there anything special that I need to do to see them?

Solution

The problem here is that on SmartOS (and other illumos variants -- as well as their proprietary Solaris cousins) the DTrace module in the JVM is lazily loaded (that is, the DOF was compiled with -x lazyload). As a result, the DTrace probes are not loaded until explicitly enabled. There are two ways to deal with this. The first is that you can tell DTrace itself to enable the specific probes in question, forcing the target process to load its probes. This requires (at least) the ID of the target process; to couch this in the example provided in the question, it would be something like:

% pfexec dtrace -ln 'hotspot*$target:::' -p `pgrep -fn "java Loop"`

This will pick up the hotspot (and hotspot_jni) USDT probes, but it still leaves using the jstack() action difficult on a machine filled with unsuspecting Java processes. (That is, this works when you want to use the USDT probes on a known process, not when you want to use the ustack helper profile all Java processes.) If this is a problem that you care about, on illumos variants (SmartOS, OmniOS, etc.) you can effectively undo the lazy loading of the DTrace probes (and stack helper) by using an audit library designed for the task. This library -- /usr/lib/dtrace/libdtrace_forceload.so and its 64-bit variant, /usr/lib/dtrace/64/libdtrace_forceload.so -- will effectively force the DTrace probes to be loaded when the process starts, giving you USDT probes and the jstack() action for all such processes. To do this for 32-bit JVMs, launch java with the LD_AUDIT_32 environment variable set:

export LD_AUDIT_32=/usr/lib/dtrace/libdtrace_forceload.so

For 64-bit JVMs:

export LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so