I have the following Spring Data JPA Repository
public interface FooRepository extends JpaRepository<Foo, String> {
@QueryHints(
value = {
@QueryHint(name = HINT_FETCH_SIZE, value = "1000"),
@QueryHint(name = HINT_CACHEABLE, value = "false"),
@QueryHint(name = HINT_FLUSH_MODE, value = "ALWAYS"),
@QueryHint(name = HINT_CACHE_MODE, value = "IGNORE"),
@QueryHint(name = HINT_READONLY, value = "true")
})
Stream<Foo> findAll();
}
called in the following method as follow
@Transactional
public void doSomething() {
AtomicInteger counter = new AtomicInteger();
try(Stream<Foo> stream = fooRepository.findAll()) {
stream.forEach(foo -> {
int i = counter.incrementAndGet();
logger.info(() -> "" + i);
});
}
}
When running this code having millions of Fooentities, this exact code throws an OutOfMemoryError. Looking at the heap dump after it crashes, I see there's a very high amount of MutableEntityEntry, Foo and EntityEntryContext$ManagedEntityImpl. All three have exact same count. On top of that, there's exactly twice that count of EntityKey. For example, I have 40k of each of the first 3 and 80k of EntityKey in the heap dump.
To make this work, I tried without success to manualy flush, clear and garbage collect as follow
@Transactional // org.springframework.transaction.annotation.Transactional
public void doSomething() {
entityManager.joinTransaction(); // properly injected through Spring DI
AtomicInteger counter = new AtomicInteger();
try(Stream<Foo> stream = fooRepository.findAll()) {
stream.forEach(foo -> {
int i = counter.incrementAndGet();
if (i % 100 == 0) {
fooRepository.flush();
entityManager.clear();
System.gc();
logger.info(() -> "flush, clear, gc");
}
logger.info(() -> "" + i);
});
}
As no reference are kept in my code to any foo entities streamed and looking to the objects in the heap dump once the error is thrown, I'm suspecting the issue is in the L1 Session cache from Hibernate even if there's a QueryHint desactivating the cache (from my understanding). It feels like only HINT_FETCH_SIZE is working in the given QueryHints on my method and I have no idea why.
FYI, I'm not using Spring Boot at all in my project. So I have the following beans in my SpringConfiguration to configure Spring Data JPA:
@Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory()
throws MalformedURLException {
HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
vendorAdapter.setDatabase(Database.POSTGRESQL);
vendorAdapter.setGenerateDdl(false);
LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
factory.setJpaVendorAdapter(vendorAdapter);
factory.setPackagesToScan(getClass().getPackage().getName());
factory.setDataSource(dataSource());
Properties jpaProperties = new Properties();
jpaProperties.setProperty(
"hibernate.physical_naming_strategy",
"my.domain.hibernate.SnakeCasePhysicalNamingStrategy");
jpaProperties.setProperty("hibernate.dialect", "org.hibernate.dialect.PostgreSQL10Dialect");
factory.setJpaProperties(jpaProperties);
return factory;
}
@Bean
public EntityManager entityManager() throws MalformedURLException {
return entityManagerFactory().getObject().createEntityManager();
}
@Bean
public PlatformTransactionManager transactionManager() throws MalformedURLException {
JpaTransactionManager txManager = new JpaTransactionManager();
txManager.setEntityManagerFactory(entityManagerFactory().getObject());
return txManager;
}
Here's the version of each
Finally found the issue which was the way the entityManager was injected in my class. Instead of adding a bean in my SpringConfiguration for it and inject it through the constructor, you must use @PersistenceContext on the field declaration in your class.
Here's the working code:
@PersistenceContext
private EntityManager entityManager;
[...]
@Transactional // org.springframework.transaction.annotation.Transactional
public void doSomething() {
entityManager.joinTransaction();
AtomicInteger counter = new AtomicInteger();
try(Stream<Foo> stream = fooRepository.findAll()) {
stream.forEach(foo -> {
int i = counter.incrementAndGet();
if (i % 100 == 0) {
entityManager.flush();
entityManager.clear();
logger.info(() -> "flush then clear);
}
logger.info(() -> "" + i);
});
}
So doing entityManager.clear() will properly clear the L1 Session Cache as it's explained here