azureazure-eventhubazure-data-explorerazure-eventgrid

Azure Data Explorer Event Grid Data Connection


I am working on to ingest data from storage to ADX using Event Grid connection which is via Event Hubs. What confusing me is as per Microsoft documentation message size for EventGrid and Event Hub can't be more than 1MB. https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas . But this (https://learn.microsoft.com/en-us/azure/data-explorer/create-event-grid-connection?tabs=portal-adx%2Cazure-blob-storage) documentation about creating Event Grid connection to ingest data in ADX via Event Hubs, says file size can go up to 6GB.

My question is- #1 How come the message size limit is different for ADX? #2 Is there a setting or configuration in Event Hub which tells what would be published? like if its a blob, message size limit is increased? #3 Is it like under the hood ADX consume messages in stream/batches which allows it to go upto 6GB?


Solution

  • My understanding is that the Event Hub would just contain the message like: "there is a new file being stored" in Json format. This message is triggered by an Event Subscription ob Blob storage.

    Then in ADX, you are defining a data connection. Based on the Event Hub which is specified in the connection, the new file will be ingested.

    enter image description hereIn this scenario, the Event Hub messages have only a few bytes. The content itself is stored in Blob, so the ingestion is not depending on the Event hub message size.