javaarrayshibernatelogginglog4j

Out of memory Requested array size exceeds VM limit Error occuring while saving a Byte array to DB


java.lang.OutOfMemoryError: Requested array size exceeds VM limit  
   at java.base/java.util.Arrays.copyOf(Arrays.java:3745 undefined)  
   at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:169 undefined)  
   at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:772 undefined)  
   at java.base/java.lang.StringBuilder.append(StringBuilder.java:248 undefined)  
   at java.base/java.util.Arrays.toString(Arrays.java:4988 undefined)  
   at org.hibernate.type.descriptor.java.PrimitiveByteArrayTypeDescriptor.extractLoggableRepresentation(PrimitiveByteArrayTypeDescriptor.java:63 undefined)  
   at org.hibernate.type.descriptor.java.PrimitiveByteArrayTypeDescriptor.extractLoggableRepresentation(PrimitiveByteArrayTypeDescriptor.java:26 undefined)  
   at org.hibernate.type.descriptor.sql.BasicBinder.bind(BasicBinder.java:70 undefined)  
   at org.hibernate.type.descriptor.converter.AttributeConverterSqlTypeDescriptorAdapter$1.bind(AttributeConverterSqlTypeDescriptorAdapter.java:88 undefined)  
   at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:280 undefined)  
   at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:275 undefined)  
   at org.hibernate.type.AbstractSingleColumnStandardBasicType.nullSafeSet(AbstractSingleColumnStandardBasicType.java:39 undefined)  
   at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate(AbstractEntityPersister.java:2879 undefined)  
   at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3176 undefined)  
   at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3699 undefined)  
   at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:90 undefined)  
   at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:604 undefined)  
   at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:478 undefined)  
   at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:356 undefined)  
   at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:50 undefined)  
   at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1418 undefined)  
   at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1504 undefined)  
   at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1561 undefined)  
   at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1529 undefined)  
   at org.hibernate.query.Query.getResultList(Query.java:168 undefined)  
   at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getResultList(CriteriaQueryTypeQueryAdapter.java:76 undefined) 

The above error is occurring while saving the below entity

@Table(name = "FILE_DATA")
public class FileData
{

    @Id
    @Column(name= "ID")
    private Long id;
    

    @Convert(converter = CustomArrayCompressionConverter.class)
    @Column(name = "CONTENT")
    private byte[] content;
}

Db Datatypes for TABLE [FILE_DATA] - [ID] [numeric](10) NOT NULL PRIMARY KEY, [CONTENT] [varbinary](max) NOT NULL

org.hibernate.type.descriptor.java.PrimitiveByteArrayTypeDescriptor.extractLoggableRepresentation(PrimitiveByteArrayTypeDescriptor.java:63) 

Here the huge byte array is being converted to String. So I need to exclude this from logging.

Issue was occurring while saving a 2 GB byte array as 2 parts of 1.7GB and 300MB. Heap size is set to 14GB.

I tried using regex filter in Log4j properties, it omits the byte array from Log file, but it's still converting this byte array to string.

I need to omit this byte array or give a custom name like 'Huge array logging' while logging byte array/

I've seen a solution using custom UserType in Hibernate trace values of statement parameters except blobs, but it cannot be used as I am using a custom AttributeConverter (CustomArrayCompressionConverter - to compress and decompress before inserting to db)

Is there any other approach to achieve this?


Solution

  • You need to turn of logging of SQL parameters.

    Set org.hibernate.orm.jdbc.bind logger to something above TRACE.

    It is possible to override extractLoggableRepresentation for PrimitiveByteArrayTypeDescriptor by registering custom version of PrimitiveByteArrayTypeDescriptor in custom dialect.

    Note that class was renamed to PrimitiveByteArrayJavaType in latest (mine is 6.5.2) version of hibernate-core.