We are using a Class that implements Cache<K, V>. The cached stored in a Hazelcast IMap<InternalKey<K, C>, V>. There's a "remove" logic that removes by an InternalKey of <String, String>.
K, C and V are all Strings.
public class HazelcastMapJCacheAdapter<K, C, V> implements Cache<K, V> {
private IMap<InternalKey<K, C>, V> cachedData;
public void removeEntriesByKeyAndContext(BiPredicate<K, C> removeCondition) {
Predicate<InternalKey<K, C>, V> predicate = (Predicate<InternalKey<K, C>, V> & Serializable)
mapEntry -> (removeCondition.test(mapEntry.getKey().getOriginalKey(), mapEntry.getKey().getContext()));
cachedData.removeAll(predicate);
}
}
I keep getting a HazelcastSerializationException which I can't understand. What can go wrong with String serialisation? How can I properly log this scenario in order to get more data (the argument I have at hand is a Predicate which is a function interface...)? I fail to reproduce this scenario on my development environment, so debugging it is a problem.
Thanks
2020-03-23 02:07:34 WARN | [10.212.179.245]:5701 [MyApp] [3.12.5] Error while logging processing event
com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.spi.impl.operationservice.impl.operations.PartitionIteratingOperation'
at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:82) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:157) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:133) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:124) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.OutboundOperationHandler.send(OutboundOperationHandler.java:56) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeRemote(Invocation.java:656) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:631) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:592) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:256) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.invokeOnAllPartitions(InvokeOnPartitions.java:121) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.invokeAsync(InvokeOnPartitions.java:99) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.invoke(InvokeOnPartitions.java:88) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnAllPartitions(OperationServiceImpl.java:385) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.proxy.MapProxySupport.removeAllInternal(MapProxySupport.java:618) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.proxy.NearCachedMapProxyImpl.removeAllInternal(NearCachedMapProxyImpl.java:330) ~[hazelcast-3.12.5.jar:3.12.5]
at com.hazelcast.map.impl.proxy.MapProxyImpl.removeAll(MapProxyImpl.java:285) ~[hazelcast-3.12.5.jar:3.12.5]
at com.myapp.cache.impl.HazelcastMapJCacheAdapter.removeEntriesByKeyAndContext(HazelcastMapJCacheAdapter.java:578) ~[myapp-distributed-cache-impl.jar:?]
Here is a summary of requirements for removeAll (and any other kind of server-side processing like executeOnKey etc):
Predicate (or EntryProcessor etc) must be serializable and available on server classpathMy guess is that in your case the BiPredicate<K, C> removeCondition is not serializable; however it has to be serialized as part of the lambda you pass to removeAll. For example consider this code:
class Scratch {
public static void main(String[] args) {
remove((x, y) -> true);
}
private static void remove(BiPredicate<String, String> predicate) {
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
// put 100 items in the map
IMap<String, String> map = hz.getMap("map");
for (int i = 0; i < 100; i++) {
map.put("" + i, "" + i);
}
map.removeAll((Predicate<String, String> & Serializable)
entry -> predicate.test(entry.getKey(), entry.getValue()));
// now size is 0
System.out.println("Map size after removeAll " + map.size());
}
}
The above code will be executed without any issues. The moment you add another HazelcastInstance in the cluster (just add Hazelcast.newHazelcastInstance() in remove method body), the Predicate instance must be serialized and sent over the network between the 2 cluster members. This fails because the predicate argument is not Serializable but it is part of the lambda Predicate passed as argument to removeAll. The solution is to make sure everything referenced in your lambda is serializable. So in the above example the fix is to update main method like this:
public static void main(String[] args) {
remove((BiPredicate<String, String> & Serializable) (x, y) -> true);
}
On a general note, exercise caution when using lambdas because it is easy to accidentally capture outer class fields and in that case you will need to address serializability of containing class and your lambda's serialized form size will balloon unexpectedly. Quoting https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html#serialization "You can serialize a lambda expression if its target type and its captured arguments are serializable. However, like inner classes, the serialization of lambda expressions is strongly discouraged."