[SOLVED] Apache Ignite Cache eviction still in memory

Apache Ignite Cache eviction still in memory

During a stability test of our apache ignite cluster we got a memory related problem, where the used memory heap space increased to 100% and didnt go down like we expected. This is what we did:

Created a cache with eviction policy to FifoEvictionPolicy( max: 10000, batchSize:100) 20 simultaneous threads that executed the following scenario over and over again for a couple of hours: Added a unique entry to the cache and then fetched the value to verify that it was added. This scenario created about 2.3 million entries during the test.

Our expectation was due to our quite restricted eviction policy at maximum 10000 entries, the memory should been stabilized. However, the memory just kept rising until it reached max heap size. See attached memory graph:

enter image description here

Our question is: Why is the memory used by the entries still allocated, even though eviction is done?

One thing to add to this, is that we executed the same test but with a deletion of the entry after that we added it. The memory was now stable:

enter image description here

Update with testcase and comment.

Below you will find a simple junit test to prove the memory leak. @a_gura seems to be correct - if we disable the ExpiryPolicy things work as expected. But if we enable ExpiryPolicy the heap seems to get filled up within the ExpiryPolicy-duration. Testcase:

public class IgniteTest {
  String cacheName = "my_cache";
  @Test
  public void test() throws InterruptedException {    
    IgniteConfiguration configuration = new IgniteConfiguration();
    Ignite ignite = Ignition.start(configuration);
    //create a large string to use as test value. 
    StringBuilder testValue = new StringBuilder();
    for (int i = 0; i < 10*1024; i ++) {
      testValue.append("a");
    }    
    CacheConfiguration cacheCfg = new CacheConfiguration();
    cacheCfg.setName(cacheName);
    cacheCfg.setEvictionPolicy(new FifoEvictionPolicy<>(10_000, 100));
    Duration duration = new Duration(TimeUnit.HOURS, 12);    
   cacheCfg.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(duration));
    cacheCfg.setCacheMode(CacheMode.LOCAL);
    cacheCfg.setBackups(0);
    Cache<Object, Object> cache = ignite.getOrCreateCache(cacheCfg);
    String lastKey = "";
    for (int i = 0; i < 10_000_101; i++){
      String key = "key#"+i;
      String value = testValue + "value#"+i;
      log.trace("storing {} {}", key, value);
      if (i % 1_000 == 0) {
        log.debug("storing {}", key);
      }
      cache.put(key, value);
      lastKey = key;
      Thread.sleep(1);
    }
    String verifyKey = "key#1";
    Assert.assertThat("first key should be evicted", cache.containsKey(verifyKey), CoreMatchers.is(false));    
    Assert.assertThat("last key should NOT be evicted", cache.containsKey(lastKey), CoreMatchers.is(true));    
    ignite.destroyCache(cacheName);    
  }
}

Solution

This has been fixed in Ignite 1.8: https://issues.apache.org/jira/browse/IGNITE-3948.

Credit to @a_gura who filed the bug and to the development team.