I'm experimenting with OGG BD (v12.3.2.1.1) to propagate all changes from an Oracle DB (11.2.0.4) to Kafka topics.
I got the basic setup all working, and INSERT and DELETE to the schemas/tables I'm monitoring are captured and produce Kafka messages onto the configured topics (1 topic per table). So far everything as expected.
But UPDATE operations on the database do not work. They are not producing any error, but simply no corresponding messages appears on the topic. I've verified with the logdump utility that the UPDATE is actually propagated into the trail file, but the kafkahandler does not pick it up.
Any ideas what might be wrong? Configurations are below
OGG PRM file
SETENV (TZ = "UTC")
REPLICAT kafkajsn
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafkajsn.props
REPORTCOUNT EVERY 1 MINUTES, RATE
REPERROR DEFAULT DISCARD
APPLYNOOPUPDATES
ALLOWNOOPUPDATES
GETTRUNCATES
GETUPDATEBEFORES
DDL INCLUDE MAPPED
GROUPTRANSOPS 100
-- all tables have an 'ID' column
MAP *.*, TARGET *.*, KEYCOLS(ID);
Kafkahandler parameter file
gg.handlerlist = kafkahandler
gg.handler.kafkahandler.type=kafka
gg.handler.kafkahandler.KafkaProducerConfigFile=kafka_producer.properties
gg.handler.kafkahandler.topicMappingTemplate=${schemaName}_${tableName}
gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}
gg.handler.kafkahandler.SchemaTopicName=SCHEMA_CHANGES
gg.handler.kafkahandler.mode=op
gg.handler.kafkahandler.format=json_row
gg.handler.kafkahandler.format.includePrimaryKeys=true
gg.classpath=dirprm/:/etc/kafka/conf:/usr/hdp/current/kafka-broker/libs/*
javawriter.bootoptions=-Xmx1g -Xms1g -Djava.class.path=ggjava/ggjava.jar
The logdump of a trail file for the REPLICAT (this was an UPDATE ... SET comment = 'foo122' WHERE id=...)
2019/05/04 21:10:02.000.601 GGSUnifiedUpdate Len 30 RBA 64926561
Name: ORDER (TDR Index: 19)
After Image: Partition 12 G s
0c00 0000 0400 0800 0000 0400 666f 6f31 0400 0a00 | ............foo1....
0000 0600 666f 6f31 3232 | ....foo122
Before Image Len 16 (x00000010)
BeforeColumnLen 12 (x0000000c)
Column 4 (x0004), Len 8 (x0008)
After Image Len 14 (x0000000e)
Column 4 (x0004), Len 10 (x000a)
EDIT: error message
I was wrong when I said it didn't produce an error message, I just didn't see it. In the discard file (.dsc) I get
Operation failed at seqno 45 rba 108411
Discarding record on action DISCARD on error 0
Problem replicating ORDER to ORDER
Mapping problem with unified update record (target format)...
*
TRANSACTION_COMMENT = folkjhlkjhlkjhkljl jk lj5345u
000000: 66 6f 6c 6b 6a 68 6c 6b 6a 68 6c 6b 6a 68 6b 6c |folkjhlkjhlkjhkl|
000010: 6a 6c 20 6a 6b 20 6c 6a 35 33 34 35 75 |jl jk lj5345u |
TRANSACTION_COMMENT = folkjhlkjhlkjhkljl jk ljjhlkjhlkh5345u
000000: 66 6f 6c 6b 6a 68 6c 6b 6a 68 6c 6b 6a 68 6b 6c |folkjhlkjhlkjhkl|
000010: 6a 6c 20 6a 6b 20 6c 6a 6a 68 6c 6b 6a 68 6c 6b |jl jk ljjhlkjhlk|
000020: 68 35 33 34 35 75 |h5345u |
I found out what the problem was. I was missing the step to ADD TRANDATA to all the tables in the source database. I had done this before, but during testing I had re-setup the whole schema and of course those setting were missing.
After applying it to all tables, I now start seeing the updates coming in as Kafka messages.
I still need to find out how to produce full before/after images in the Kafka message, currently it only logs the ID columns and the modified ones.