azurewatermarkazure-stream-analyticsstream-analytics

Stream Analytics not processing instantly, constant 5 second watermark


We are trying to process event driven icons on a dashboard (on/off/active) with IoT hub, stream analytics, and table storage. I cannot get the actual process to work under 6 seconds. Watermark for stream analytics is a constant 5 seconds. The website claims instant processing. https://azure.microsoft.com/en-us/services/stream-analytics/

I have setup stream analytics with a very simple query of storing the enqueued time into table storage. I made sure the event ordering times and out of order messages was all set to 0 seconds and drop. I did this with a new stream analytics deployment. Also used the max allowed SU of 6. I set the batch output to 1.

select IoTHub.ConnectionDeviceId as PartitionKey, IoTHub.ConnectionDeviceId as RowKey, IoTHub.EnqueuedTime As Value
into Output
from Input

I expect total processing to be under 1.5 seconds and the watermark to be under 100s of milliseconds.


Solution

  • The following alerts are recommended for monitoring the performance of your Stream Analytics job. These metrics should be evaluated every minute over the last 5-minute period.

    enter image description here

    Try increasing the number of SUs or parallelizing your query.

    For more information on SUs, see Understand and adjust Streaming Units.

    Hope this helps.