addShutdownHook says:
In rare circumstances the virtual machine may abort, that is, stop running without shutting down cleanly. This occurs when the virtual machine is terminated externally, for example with the SIGKILL signal on Unix or the TerminateProcess call on Microsoft Windows. The virtual machine may also abort if a native method goes awry by, for example, corrupting internal data structures or attempting to access nonexistent memory. If the virtual machine aborts then no guarantee can be made about whether or not any shutdown hooks will be run.
When the Java heap runs out of memory and throws an OutOfMemoryError, does that cause the JVM to abort? Do heap OOMs cause native methods to go awry or corrupt internal data structures? Or are heap OOM errors recoverable enough for the shutdown hooks to be executed?
When the Java heap runs out of memory, the JVM will throw an OutOfMemoryError and may abort. Whether the JVM aborts or not depends on how the OutOfMemoryError is handled by the application. If the error is not caught and handled by the application, the JVM may choose to abort.
It is important to note that OutOfMemoryError is a runtime exception, so it is not required for the application to explicitly catch and handle it. However, if the application does catch and handle the error, it may be able to recover from the situation and continue running.
Whether or not the OutOfMemoryError causes native methods to go awry or corrupt internal data structures depends on how the error is handled by the application and by the JVM itself. In general, it is best to avoid letting the heap run out of memory, as this can cause unpredictable behavior.
If the JVM does not abort, it is possible for the shutdown hooks to be executed. Shutdown hooks are a mechanism for running specific cleanup code when the JVM is shutting down. However, if the JVM does abort, the shutdown hooks may not be executed.