I'm looking for a streaming server with the message persistence guarantee, i.e. where the messages published by producers are guaranteed to be durably stored before the server acknowledges publishing to the producer.
My use case requires that we reduce the possibility of losing any produced messages. Producers are able to replay messages if required but they need to be sure that the ACKed message is durably persisted and will be delivered by the streaming server to the consumers.
NATS Streaming server seems to do something along those lines, but the docs for clustering and fault tolerance don't make it very clear what persistence guarantee is provided in each case. The doc on producer integration confirms that the server will actively ACK the published messages, either synchronously or via callback, but it does not make it clear if the ACK means that the message was durably stored at this point or not yet.
The doc on store configuration, specifically SQL options briefly mentions that the ACK from the server means a durable storage guarantee, but it's not clear still how exactly that applies in cases of Clustering and Fault Tolerance and different persistence backends (files or SQL).
NATS Streaming will have persisted the message before sending the publisher ACK back. The store implementations (filestore/SQL) may use some caching, but regardless, the writes are sync'ed (unless disabled) before the ACK is sent back.
However, in cluster mode, the filestore sync'ing is disabled because we rely on the fact that the data is replicated to each node of the cluster and so you would need multiple failures at once to lose the message. (note that there is an option for file store implementation to perform auto-sync at regular interval: see auto_sync here