Is it possible/reasonable to use OpenJ9's class sharing feature to reduce the memory usage of single instances of different applications?
More details:
It seems like the feature is designed for reducing memory usage when running multiple instances of the same application on the same machine (e.g. Kubernetes node). Is that correct?
What I would like to achieve is that applications that coincidentally are started on the same Kubernetes node use the same cache. Since the applications are using almost the same libraries, there should be a nontrivial amount of data that can be shared. However, what I experience is caches with a high percentage of stale classes and even caches becoming 100% full even though I use soft and hard limits of 1GB.
Specifically, I have tried using java -Xshareclasses:cacheDir=/openj9cache,groupAccess,name=somename,nonfatal,persistent,verbose -XX:SharedCacheHardLimit=1g -Xscmx1g <APP> for different deployments on the same Kubernetes node. Depending on what applications I run, I see cache statistics like:
cache layer = 0
cache size = 1073741216
softmx bytes = 1073741824
free bytes = 0
Reserved space for AOT bytes = -1
Maximum space for AOT bytes = -1
Reserved space for JIT data bytes = -1
Maximum space for JIT data bytes = -1
Metadata bytes = 937848376
Metadata % used = 87%
Class debug area size = 85868544
Class debug area used bytes = 4246022
Class debug area % used = 4%
ROMClass bytes = 36481772
AOT bytes = 12671632
JIT data bytes = 506596
Zip cache bytes = 0
Startup hint bytes = 360
Data bytes = 363936
# ROMClasses = 45685
# AOT Methods = 2997
# Classpaths = 30076
# URLs = 0
# Tokens = 0
# Zip caches = 0
# Startup hints = 3
# Stale classes = 40496
% Stale classes = 88%
Cache is 100% soft full
cache layer = 0
cache size = 1073741216
softmx bytes = 1073741824
free bytes = 0
Reserved space for AOT bytes = -1
Maximum space for AOT bytes = -1
Reserved space for JIT data bytes = -1
Maximum space for JIT data bytes = -1
Metadata bytes = 935010252
Metadata % used = 87%
Class debug area size = 85868544
Class debug area used bytes = 4745600
Class debug area % used = 5%
ROMClass bytes = 40151980
AOT bytes = 11919936
JIT data bytes = 426448
Zip cache bytes = 0
Startup hint bytes = 120
Data bytes = 363936
# ROMClasses = 38554
# AOT Methods = 2950
# Classpaths = 22680
# URLs = 0
# Tokens = 0
# Zip caches = 0
# Startup hints = 1
# Stale classes = 6354
% Stale classes = 16%
Cache is 100% soft full
base address = 0x00007F3D8C059000
end address = 0x00007F3DCC000000
allocation pointer = 0x00007F3D8F10BB80
cache layer = 0
cache size = 1073741216
softmx bytes = 1073741824
free bytes = 706740256
Reserved space for AOT bytes = -1
Maximum space for AOT bytes = -1
Reserved space for JIT data bytes = -1
Maximum space for JIT data bytes = -1
Metadata bytes = 199367772
Metadata % used = 54%
Class debug area size = 85868544
Class debug area used bytes = 6499570
Class debug area % used = 7%
ROMClass bytes = 51063680
AOT bytes = 29461056
JIT data bytes = 875612
Zip cache bytes = 0
Startup hint bytes = 360
Data bytes = 363936
# ROMClasses = 30778
# AOT Methods = 6624
# Classpaths = 8349
# URLs = 0
# Tokens = 0
# Zip caches = 0
# Startup hints = 3
# Stale classes = 3274
% Stale classes = 10%
Cache is 34% soft full
It is the best practice to use the same shared cache for multiple instances of the same (or similar) application. You may not get the (most) memory savings if you are sharing the same cache among different applications, as the shared cache contains classes that are stored by other apps which are not needed by the current app.
OpenJ9 is doing timestamp checking on jars on the class path. It makes sure the class on the class path has not been updated since it is stored into the cache, so that we can safely return the cached one to the class loader. Once an update is found, the original cached class is marked as stale.
Using sub-option "bootClassesOnly" turns on class sharing for bootstrap classes only. No one is updating things on the bootstrap class path, so you see no stale classes.
I see there are thousands of class paths in the cache statistics. Did things on these class paths get updated or are they generated at runtime ? You can use -Xshareclasses sub-option "printStats=classpath" to show all the class paths. If you are sure the classes are not updated between runs, you can use -Xshareclasses sub-option "noTimestampChecks" to turn off timestamp checking.