I had implemented my own serializer in Java. Lets call this as abcSerializer. The object that I am trying to serialize is abc, which is a Google Protocol Buffer class.
I am trying to use kryo frame work to serialize this object. After bit of research and reading over google, i decided that to go ahead with kryo serializer. I didnt specify any serializer per se, so i assume kryo pick a default serializer.
public class abcSerializer implements AttributeSerializer <abc> {
public abcSerializer() {
kryo = new Kryo();
}
public static Kryo getKryo() {
return kryo;
}
@Override
public abc read(byte[] buffer) {
abc xyz = null;
ByteArrayInputStream abcsStream = new ByteArrayInputStream(buffer);
Input abcsStreamInput = new Input(abcsStream);
xyz = getKryo().readObject(abcsStreamInput, abc.class);
return xyz;
}
@Override
public void write(byte[] buffer, abc abc) {
ByteArrayOutputStream abcStream = new ByteArrayOutputStream();
Output abcOutput = new Output(abcStream);
getKryo().writeObject(abcOutput, abc);
abcOutput.toBytes()
}
}
When I do writeObject, all is well. However the problem comes when I do readObject. Kyro throws up the below exception.
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.util.Collections$Unmodifiabljava.lang.IllegalStateException: Encountered error in deserializer [null value returned]. Check serializer compatibility.eRandomAccessList
The above exception is pretty much self explanatory.
The kryo documentation is as below. ""Serializers for a specific type use Java code to create a new instance of that type. Serializers such as FieldSerializer are generic and must handle creating a new instance of any class. By default, if a class has a zero argument constructor then it is invoked via ReflectASM or reflection, otherwise an exception is thrown. If the zero argument constructor is private, an attempt is made to access it via reflection using setAccessible. If this is acceptable, a private zero argument constructor is a good way to allow Kryo to create instances of a class without affecting the public API.""
Now, I have two questions.
1) the google protocol buffer generated class does have a no arg constructor. However this seems to be a private. Is this a problem and the root cause for the above kryo exception ?
2) If so, how to move forward on the above issue ? I mean, how can i write a serializer of my own and still use to serialize the google protocol buffer object data ?
How's this work for you?
/**
* This lets Kryo serialize protobufs more efficiently.
*/
public class ProtobufKryo<P extends GeneratedMessage> extends Serializer<P> {
protected final Method parser;
public ProtobufKryo(Class<P> theClass) {
try {
parser = theClass.getDeclaredMethod("parseFrom", InputStream.class);
parser.setAccessible(true);
} catch (NoSuchMethodException e) {
throw new IllegalArgumentException(theClass.toString() + " doesn't have parser");
}
}
@Override
public void write(Kryo kryo, Output output, P generatedMessage) {
try {
generatedMessage.writeTo(output);
} catch (IOException e) {
// This isn't supposed to happen with a Kryo output.
throw new RuntimeException(e);
}
}
@Override
public P read(Kryo kryo, Input input, Class<P> gmClass) {
try {
return (P)parser.invoke(null, input);
} catch (InvocationTargetException | IllegalAccessException e) {
// These really shouldn't happen
throw new IllegalArgumentException(e);
}
}
}
OK,, some explanation....
When Kryo encounters an object of a class it doesn't recognize, it falls back to Java serialization. Not every efficient, and sometimes doesn't work.
(Well, I confess that the above may not always be true. It may be part of Kryo configuration. It is true in the environment I work in.)
You can tell it to use its own serialization for specific classes, but sometimes you can do better by creating a custom serializer for specific classes.
The above leverages protobufs' existing serialization within Kryo. Basically, it uses the existing protobuf writeTo() and parseFrom() to handle serialization within Kryo. You'd register the above class to serialize each of your protobuf classes. (Protobuf classes extend GeneratedMessage.)
Writing out the object just uses the normal protobuf writeTo() method. Reading back in the protobuf uses the classes parseFrom() method, which is found via reflection in the constructor.
So, you'd configure your serializer with something like:
Kryo k = new Kryo();
k.addDefaultSerializer(MyProtobuf.class, ProtobufKryo.class);
k.addDefaultSerializer(MyOtherProtobuf.class, ProtobufKryo.class);
etc.