avrostreamsetsdivolte

Cant find Avro schema in divolte and kafka docker


I have 3 dockers kafka, divolt and streamsets (https://github.com/divolte/docker-divolte) started by compose-up. I want to convert the topic messages to avro files. I created the pipeline in streamset and paste the avro schema, but got an error:

com.streamsets.pipeline.api.base.OnRecordErrorException: KAFKA_37 - Cannot parse record from message 'divolte::3::0': java.io.IOException: Invalid int encoding
        at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource$MultiTopicCallable.createRecord(MultiKafkaSource.java:192)
        at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource$MultiTopicCallable.sendBatch(MultiKafkaSource.java:158)
        at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource$MultiTopicCallable.call(MultiKafkaSource.java:135)
        at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource$MultiTopicCallable.call(MultiKafkaSource.java:79)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Invalid int encoding

I red that problem is in incorrect avro schema. Could you tell where I can find the correct avro schema for this? I cant find it in dockers and github.


Solution

  • Looks like it might be in the Divolte GitHub repo, at https://github.com/divolte/divolte-schema/blob/master/src/main/resources/DefaultEventRecord.avsc