I'm trying to create a simple flow with Spring Integration and Project Reactor, where I consume records with Reactor Kafka, passing them to a channel that from there it will produce messages into another topic with Reactor Kafka.
The consuming flow is:
@Service
public class ReactiveConsumerService {
public ReactiveKafkaConsumerTemplate<String, String> reactiveKafkaConsumerTemplate;
@Qualifier("directChannel")
@Autowired
public MessageChannel directChannel;
public ReactiveConsumerService(ReactiveKafkaConsumerTemplate<String, String> reactiveKafkaConsumerTemplate) {
this.reactiveKafkaConsumerTemplate = reactiveKafkaConsumerTemplate;
}
@Bean
public IntegrationFlow readFromKafka() {
return IntegrationFlows.from(reactiveKafkaConsumerTemplate.receiveAutoAck()
.map(GenericMessage::new))
.<ConsumerRecord<String, String>, String>transform(ConsumerRecord::value)
.<String, String>transform(String::toUpperCase)
.channel(directChannel)
.get();
}
}
And the producing flow is:
@Service
public class ReactiveProducerService {
private final ReactiveKafkaProducerTemplate<String, String> reactiveKafkaProducerTemplate;
@Qualifier("directChannel")
@Autowired
public MessageChannel directChannel;
public ReactiveProducerService(ReactiveKafkaProducerTemplate<String, String> reactiveKafkaProducerTemplate) {
this.reactiveKafkaProducerTemplate = reactiveKafkaProducerTemplate;
}
@Bean
public IntegrationFlow kafkaProducerFlow() {
return IntegrationFlows.from(directChannel)
.handle(s -> reactiveKafkaProducerTemplate.send("topic2", s.getPayload().toString()))
.get();
}
}
I'd like to know how and where exactly should I perform the subscription.
I've added a .subscripe() and it still doesn't work:
2022-01-25 20:36:59.570 INFO 1804 --- [ration-sample-1] o.a.kafka.common.utils.AppInfoParser : App info kafka.consumer for consumer-reactive-kafka-spring-integration-sample-1 unregistered
2022-01-25 20:36:59.573 ERROR 1804 --- [oundedElastic-1] reactor.core.publisher.Operators : Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.IllegalStateException: No subscriptions have been created
Caused by: java.lang.IllegalStateException: No subscriptions have been created
at reactor.kafka.receiver.ReceiverOptions.subscriber(ReceiverOptions.java:423) ~[reactor-kafka-1.3.9.jar:1.3.9]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Assembly trace from producer [reactor.core.publisher.FluxPeekFuseable] :
reactor.core.publisher.Flux.doOnRequest
Caused by: java.lang.IllegalStateException: No subscriptions have been created
reactor.kafka.receiver.internals.ConsumerHandler.receive(ConsumerHandler.java:110)
Error has been observed at the following site(s):
*________Flux.doOnRequest ? at reactor.kafka.receiver.internals.ConsumerHandler.receive(ConsumerHandler.java:110)
|_ Flux.filter ? at reactor.kafka.receiver.internals.DefaultKafkaReceiver.lambda$receiveAutoAck$6(DefaultKafkaReceiver.java:70)
|_ Flux.publishOn ? at reactor.kafka.receiver.internals.DefaultKafkaReceiver.lambda$receiveAutoAck$6(DefaultKafkaReceiver.java:71)
|_ Flux.map ? at reactor.kafka.receiver.internals.DefaultKafkaReceiver.lambda$receiveAutoAck$6(DefaultKafkaReceiver.java:72)
*______________Flux.using ? at reactor.kafka.receiver.internals.DefaultKafkaReceiver.lambda$withHandler$19(DefaultKafkaReceiver.java:137)
*__________Flux.usingWhen ? at reactor.kafka.receiver.internals.DefaultKafkaReceiver.withHandler(DefaultKafkaReceiver.java:129)
|_ ? at reactor.kafka.receiver.internals.DefaultKafkaReceiver.receiveAutoAck(DefaultKafkaReceiver.java:68)
|_ ? at reactor.kafka.receiver.KafkaReceiver.receiveAutoAck(KafkaReceiver.java:124)
|_ Flux.concatMap ? at org.springframework.kafka.core.reactive.ReactiveKafkaConsumerTemplate.receiveAutoAck(ReactiveKafkaConsumerTemplate.java:69)
|_ Flux.map ? at reactor.kafka.spring.integration.samples.service.ReactiveConsumerService.readFromKafka(ReactiveConsumerService.java:38)
|_ Flux.from ? at org.springframework.integration.channel.FluxMessageChannel.subscribeTo(FluxMessageChannel.java:118)
|_ Flux.delaySubscription ? at org.springframework.integration.channel.FluxMessageChannel.subscribeTo(FluxMessageChannel.java:119)
|_ Flux.publishOn ? at org.springframework.integration.channel.FluxMessageChannel.subscribeTo(FluxMessageChannel.java:120)
|_ Flux.doOnNext ? at org.springframework.integration.channel.FluxMessageChannel.subscribeTo(FluxMessageChannel.java:121)
Original Stack Trace:
at reactor.kafka.receiver.ReceiverOptions.subscriber(ReceiverOptions.java:423) ~[reactor-kafka-1.3.9.jar:1.3.9]
at reactor.kafka.receiver.internals.ConsumerEventLoop$SubscribeEvent.run(ConsumerEventLoop.java:207) ~[reactor-kafka-1.3.9.jar:1.3.9]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.4.14.jar:3.4.14]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.4.14.jar:3.4.14]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
2022-01-25 20:36:59.772 INFO 1804 --- [ main] o.s.b.web.embedded.netty.NettyWebServer : Netty started on port 8090
2022-01-25 20:36:59.853 INFO 1804 --- [ main] o.a.c.impl.engine.AbstractCamelContext : Routes startup summary (total:0 started:0)
2022-01-25 20:36:59.853 INFO 1804 --- [ main] o.a.c.impl.engine.AbstractCamelContext : Apache Camel 3.12.0 (camel-1) started in 149ms (build:84ms init:59ms start:6ms)
2022-01-25 20:36:59.866 INFO 1804 --- [ main] ReactorKafkaSpringIntegrationApplication : Started ReactorKafkaSpringIntegrationApplication in 4.246 seconds (JVM running for 4.616)
The sample code:
@Service
public class ReactiveProducerService {
private final ReactiveKafkaProducerTemplate<String, String> reactiveKafkaProducerTemplate;
@Qualifier("directChannel")
@Autowired
public MessageChannel directChannel;
public ReactiveProducerService(ReactiveKafkaProducerTemplate<String, String> reactiveKafkaProducerTemplate) {
this.reactiveKafkaProducerTemplate = reactiveKafkaProducerTemplate;
}
@Bean
public IntegrationFlow kafkaProducerFlow() {
return IntegrationFlows.from(directChannel)
.handle(s -> reactiveKafkaProducerTemplate.send("topic2", s.getPayload().toString()).subscribe(System.out::println))
.get();
}
}
The subscription to the reactiveKafkaConsumerTemplate happens immediately when the endpoint for the .<ConsumerRecord<String, String>, String>transform(ConsumerRecord::value) is started automatically by the application context.
See this one as an alternative:
/**
* Represent an Integration Flow as a Reactive Streams {@link Publisher} bean.
* @param autoStartOnSubscribe start message production and consumption in the flow,
* when a subscription to the publisher is initiated.
* If this set to true, the flow is marked to not start automatically by the application context.
* @param <T> the expected {@code payload} type
* @return the Reactive Streams {@link Publisher}
* @since 5.5.6
*/
@SuppressWarnings(UNCHECKED)
protected <T> Publisher<Message<T>> toReactivePublisher(boolean autoStartOnSubscribe) {
Although I think you mean the subscription on the outbound side. It is not clear from your question, but that reactiveKafkaProducerTemplate has a contract like:
public Mono<SenderResult<Void>> send(String topic, V value) {
So, you need to subscribe to that returned Mono to initiate a process.
NOTE: you have messed arguments for that send() as well. Didn't you mean this instead: reactiveKafkaProducerTemplate.send("test", "topic2") ?
To make it subscribing to that Mono, you just need to do that yourself in that handle():
.handle(s -> reactiveKafkaProducerTemplate.send("topic2", "test").subscribe())
UPDATE 2
The error like java.lang.IllegalStateException: No subscriptions have been created from the reactor.kafka.receiver.ReceiverOptions.subscriber() means that you didn't assign topic, patterns or partitions to listen to.
See ReceiverOptions.subscription() or ReceiverOptions.assignment().