Method for finding memory leak in large Java heap dumps

I have to find a memory leak in a Java application. I have some experience with this but would like advice on a methodology/strategy for this. Any reference and advice is welcome.

About our situation:

1. Heap dumps are larger than 1 GB
2. We have heap dumps from 5 occasions.
3. We don't have any test case to provoke this. It only happens in the (massive) system test environment after at least a weeks usage.
4. The system is built on a internally developed legacy framework with so many design flaws that they are impossible to count them all.
5. Nobody understands the framework in depth. It has been transfered to one guy in India who barely keeps up with answering e-mails.
6. We have done snapshot heap dumps over time and concluded that there is not a single component increasing over time. It is everything that grows slowly.
7. The above points us in the direction that it is the frameworks homegrown ORM system that increases its usage without limits. (This system maps objects to files?! So not really a ORM)

Question: What is the methodology that helped you succeed with hunting down leaks in a enterprise scale application?

It's almost impossible without some understanding of the underlying code. If you understand the underlying code, then you can better sort the wheat from chaff of the zillion bits of information you are getting in your heap dumps.

Also, you can't know if something is a leak or not without know why the class is there in the first place.

I just spent the past couple of weeks doing exactly this, and I used an iterative process.

First, I found the heap profilers basically useless. They can't analyze the enormous heaps efficiently.

Rather, I relied almost solely on jmap histograms.

I imagine you're familiar with these, but for those not:

jmap -histo:live <pid> > histogram.out

creates a histogram of the live heap. In a nutshell, it tells you the class names, and how many instances of each class are in the heap.

I was dumping out heap regularly, every 5 minutes, 24hrs a day. That may well be too granular for you, but the gist is the same.

I ran several different analyses on this data.

I wrote a script to take two histograms, and dump out the difference between them. So, if java.lang.String was 10 in the first dump, and 15 in the second, my script would spit out "5 java.lang.String", telling me it went up by 5. If it had gone down, the number would be negative.

I would then take several of these differences, strip out all classes that went down from run to run, and take a union of the result. At the end, I'd have a list of classes that continually grew over a specific time span. Obviously, these are prime candidates for leaking classes.

However, some classes have some preserved while others are GC'd. These classes could easily go up and down in overall, yet still leak. So, they could fall out of the "always rising" category of classes.

To find these, I converted the data in to a time series and loaded it in a database, Postgres specifically. Postgres is handy because it offers statistical aggregate functions, so you can do simple linear regression analysis on the data, and find classes that trend up, even if they aren't always on top of the charts. I used the regr_slope function, looking for classes with a positive slope.

I found this process very successful, and really efficient. The histograms files aren't insanely large, and it was easy to download them from the hosts. They weren't super expensive to run on the production system (they do force a large GC, and may block the VM for a bit). I was running this on a system with a 2G Java heap.

Now, all this can do is identify potentially leaking classes.

This is where understanding how the classes are used, and whether they should or should not be their comes in to play.

For example, you may find that you have a lot of Map.Entry classes, or some other system class.

Unless you're simply caching String, the fact is these system classes, while perhaps the "offenders", are not the "problem". If you're caching some application class, THAT class is a better indicator of where your problem lies. If you don't cache com.app.yourbean, then you won't have the associated Map.Entry tied to it.

Once you have some classes, you can start crawling the code base looking for instances and references. Since you have your own ORM layer (for good or ill), you can at least readily look at the source code to it. If you ORM is caching stuff, it's likely caching ORM classes wrapping your application classes.

Finally, another thing you can do, is once you know the classes, you can start up a local instance of the server, with a much smaller heap and smaller dataset, and using one of the profilers against that.

In this case, you can do unit test that affects only 1 (or small number) of the things you think may be leaking. For example, you could start up the server, run a histogram, perform a single action, and run the histogram again. You leaking class should have increased by 1 (or whatever your unit of work is).

A profiler may be able to help you track the owners of that "now leaked" class.

But, in the end, you're going to have to have some understanding of your code base to better understand what's a leak, and what's not, and why an object exists in the heap at all, much less why it may be being retained as a leak in your heap.