I've created a client of Apache Kafka using Kafkajs, and I'm trying to read messages from a topic in Kafka. It's working fine if I console.log(message). But I want to send message to client everytime whenever a new message is produced/written in the topic, the consumer is listening to, from producers, while keeping the connection alive.
// function, which is being called whenever it's specified route is being requested
async readMessage(req, res, next, consumer) {
const resMessage = {};
res.writeHead(200, {'Content-Type': 'text/plain'});
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
res.write(message.value.toString());
},
});
// res.send(resMessage);
}
But after I send data to the express.js server, res.write() doesn't send the data to the client (I'm using Postman as my Node.js client). How do I flush the data written in res.write(), before res.end() is invoked?
res.write() does send the data (no buffering by Express or by nodejs) as soon as you call it. I've traced through it in the debugger and seen it send the data without delay. It may be buffered briefly by the OS in service of Nagle's algorithm, but that would only be a very short delay (milliseconds). So, the data is getting sent to the client. It is much more likely that your issue is in the http client that is receiving the data.
Most http clients are built to expect request/response. Send a request, wait for the entire response, then notify the caller. So, if you want to get regular data from the http stream as it arrives, you will need a non-traditional http client that will notify you when data arrives like that. You will probably also need to invent some sort of protocol that allows the client to know when a full piece has arrived since packets can be chunked or grouped in random ways.
What might be easier than using http for this is to use an actual message-based protocol such as webSockets or socket.io that is explicitly designed for the exact kind of thing you're trying to do. The client would establish a connection to the server and then the server is free to send messages to the client at any time. webSocket or socket.io are message-based so they already do all the work for you of delineating a message, packing it together, delivering it, unpacking it and notifying the recipient that a message has arrived. This is exactly what webSocket/socket.io were designed for.
For one-way message sending (from server to client), you could also use Server Sent Events which is an extension of http.