I'm experiencing an issue with ILP ingestion delay. It seems there's a delay of 1-1.5 seconds between when data is supposed to appear in the database and when it actually does. I believe this might be related to some sort of buffering mechanism.
After
some research, I found a parameter that seems relevant: cairo.o3.min.lag. Could this be
the correct parameter to adjust to reduce the delay? Our workload typically ranges from 1
to 50 events per second, with an average of about 10 events per second.
To address the ingestion delay you're experiencing, adjusting the cairo.o3.min.lag
parameter is indeed a step in the right direction. This parameter controls the minimum lag before committing data to the database, with its default set to 1 second. If your goal is to make data appear in the database more quickly, reducing this value can help.
However, there are a few considerations to keep in mind:
Server-Side Limitations: It's important to note that the ability to control this delay might not be entirely possible from the server side. Some QuestDB clients, including the JAVA client used by the Kafka connector, has built-in auto-flushing mechanisms. Although these configurations can be modified when using the client libraries, they may not be directly exposed through connector configuration options.
Resource Considerations: Reducing the cairo.o3.min.lag to a lower value, such as 50ms,
can indeed decrease the delay. However, this comes at a cost, specifically in terms of CPU
usage, which is expected to increase. The increase in CPU usage is due to the database
committing data more frequently. For a workload where a new message arrives every 20ms to 100ms,
decreasing the cairo.o3.min.lag to 50ms is feasible. However, expect a significant
increase in CPU usage—potentially up to 20 times higher if the setting is decreased
proportionally. It's crucial to understand that the cost of processing and committing
data in QuestDB is mostly associated with the commit operation itself, rather than the
amount of data being processed in each commit.
Impact on Different Table Types: The effect of changing cairo.o3.min.lag applies to
both WAL tables and non-WAL tables, with non-WAL tables being more complicated in terms
of handling commits.
Testing and monitoring the impact of such changes on your specific environment and workload is highly recommended.