I'm encountering a timeout issue when registering a stream with Memgraph. Whenever I register a stream named "test_stream", it times out during the CHECK STREAM command. Despite the initial timeout, I can start the stream afterwards without any issues, and it runs as expected. Below are the logs I collected with the log level set to trace:
[2024-04-30 16:13:09.176][Debug][Run - memgraph] 'CREATE KAFKA STREAM test_stream TOPICS transferEdges TRANSFORM aTransferA.account_transfer_account BATCH_INTERVAL 100 BATCH_SIZE 1000 CONSUMER_GROUP mg_consumer BOOTSTRAP_SERVERS "localhost:9092"'
[2024-04-30 16:13:09.325][Debug][Run - memgraph] 'CHECK STREAM test_stream BATCH_LIMIT 1 TIMEOUT 30000'
[2024-04-30 16:13:09.517][Trace] Error message: Kafka consumer test_stream check failed: timeout reached
[2024-04-30 16:13:09.923][Trace] Reset received
Every stream registration follows this pattern of timing out initially but functioning correctly thereafter. Is there a way to bypass the CHECK STREAM command?
I guess you're creating the stream through the Memgraph Lab UI, since the CHECK STREAM query is being automatically run?
To avoid it, you can create a stream in the query execution, instead of using the Lab UI for creating the stream. You just execute the CREATE STREAM command with the existing transformation module you created (https://memgraph.com/docs/data-streams#create-a-stream) and after that you run start stream (https://memgraph.com/docs/data-streams#start-a-stream).
I am not sure what happened with the CHECK STREAM and offset that you're mentioning, so I'll check that with the team tomorrow and get back to you
I hope that at least this step of avoiding CHECK STREAM helps.
Btw., regarding the offset, it can be updated as well -
https://memgraph.com/docs/data-streams#setting-a-stream-offset