jsonazure-eventhubazure-stream-analytics

How to efficiently push compressed json data into azure event hub and proccess in azure stream analytics?


We have a backend application which has to send telemetry data to an event hub. All data have to be serialized into JSON and compressed.

Should we collect all serialized objects into a single newline-delimited JSON or it is better to use one EventData wrapper per object and send it as a batch? Probably compression will work better with newline-delimited json. But will ASA be able to process it?


Solution

  • Asa supports gzip and deflate compression. Every eventhub message can be up to 256 kb including metadata. On processing side, every message has some overhead. So, for same number of records, smaller number of eventhub messages is better. However, this usually means some buffering on the send side. Based on your overall latency requirements and memory footprint requirements on sender, you should batch multiple records into every eventhub message, with compression.