I am writing a Trident topology to process stream of data from Kafka and feed in to Redis and Cassandra. I am able to write the data in to Cassandra. Now I would like to write the same data in to Redis.
Is there a way to duplicate the tuples and branch it in to 2 flow where one goes in to Redis and another goes in to Cassandra?
For Trident you can go with smth like this:
TridentTopology topology = new TridentTopology();
Stream stream = topology.newStream("MySpout", spout);
stream.partitionPersist(...); // to Redis
stream.partitionPersist(...); // to Cassandra
So it will be saving the data from your stream to both databases in parallel.
However I would also think if such parallel thing should be done inside a single topology or if having two different topologies reading from the same topic is a better idea. Imagine Cassandra cluster goes down. In case of two topologies you'll still be able to continue saving the data to Redis. But if there's just a single topology, every tuple failed to go to Cassandra most likely will result in a FailedException to trigger replaying and every subsequent replay of a tuple will involve saving the tuple to Redis once again unnecessarily.