I am making a real-time based chat microservice and this is my first time making such application. I am kind of confused on how to store the conversations because I think storing them in a database (Cassandra or whatever) is not an optimal solution for the long run.
I've read about Apache Kafka and found that this could be a solution to my problem. Thus, I'm not quite sure if I'm there yet. I just need to know If Kafka is enough without the need of an external database (Just talking about the data related to the messages though, not the users or any other type of data I may need). Because I've read that Kafka provides in its topics the option to use a retention time of "-1" which basically means forever, but AS LONG AS THE SERVER IS RUNNING while I'm not sure I can keep the same server running forever, could someone please clarify this for me ?
read about Apache Kafka and found that this could be a solution to my problem.... need to know If Kafka is enough
For (temporary) storage of the event itself, sure. For message delivery to clients, no.
Some extreme examples come up immediately when considering using any message broker
Most importantly, chat apps are frontend applications; Kafka is a backend technology and doesn't "push" to frontend services. For mobile/browser use-cases, you wouldn't have access to embed a Kafka consumer in them. A REST API layer would have to be added to request messages, just as it would with a database.
A better design would be to generate message events rather than store message data - E.g. "At time T
, user X
send 'M'
message with content C
to location/users L
"... A topic with that data does not need to be persisted forever, just long enough to be consumed. And still, with all that information, you then need to consider topic partitioning for scalability, then you can filter and dump that into some more persistent location because your user-facing application wouldn't be consuming every single one of those events.
I'm not sure I can keep the same server running forever
Someone will have to. Use a hosted solution, if you cannot.