Drools going OutOfMemory after upgrade 6.4.0 to 8.44.2

We're upgrading a Drools Fusion application from 6.4.0 to 8.44.2. The application is managing just 1 single long running KieSession in stream mode (config.setOption(EventProcessingOption.STREAM)). Events are inserted into the session which is running in fireUntilHalt().

In 6.4.0, the same session may have run for days or weeks. We experienced that events can arrive in extreme bursts and cause OutOfMemory issues, when we just relied on the automatic eviction of Events (after 24h by our configuration). Hence we added some extra logic that checked before event inserts that not more than N events were already in the session. If this happened, we would pick the 20% oldest events and manually remove the associated facts from the session (with code like below, were ep is the entry point, and Event is our application domain event class).

for (Event e : toDelete) {
    FactHandle fh = ep.getFactHandle(e);
    if (fh != null) {
        ep.delete(fh);
    }
}

The same code runs now with 8.44.2, but from what we can tell, the memory consumption is slowly but steadily increasing. The manual deletion logic still seems to work, we see related log entries. But the session at some point goes towards OOM. The memory histogram then looks like this

In that test setup, we always checked that the number of Events stays below 11'000. If it reaches this level, the oldest 2000 are ep.delete()'d from the session. Still, the number of active events seems to ever increase. Most memory seems to be held in the TruthMaintenanceSystemRuleTerminalNodeLeftTuple class.

I must admit that I have no idea what this class is used for, and why its instances are apparently retained "forever". Any idea what could be happening here?

Because you are dealing with entry point to get fact handle and delete from, could you please confirm you delete from the correct entry point? Do you really mean entry point in drl (because session also implements entry point)? Why do you have condition at line 3 which handles the case when object was not found in the working memory, is this expected (part of the business logic) or you are just hiding wrong deletion logic (deletion from wrong entry point)?

1 for (Event e : toDelete) {
2     FactHandle fh = ep.getFactHandle(e);
3     if (fh != null) {
4         ep.delete(fh);
5     }
6 }

Could you please try other way of deletion, not to rely on the code outside of the drools working memory to keep track of objects (and storing reference to them)?

global AtomicLong maxCount

rule "maxCount"
    salience -1
when
    $new: CountedModel()
    not CountedModel(ordinal > $new.ordinal)
    $old: CountedModel()
    not CountedModel(ordinal < $old.ordinal)
then
    if ($new.ordinal - $old.ordinal >= maxCount.get()) {
        delete($old);
    }
end