Azure Event Hub listening to blob changes

I have a use case where I have a streaming pipeline streams data into azure blob storage. I want to pick the data up from the azure blob through event hub, and configure the streaming ingestion into Azure Data Explorer.

I have try to create both event grid (blob storage) on ADX cluster, and the event hub namespace with one event hub pointing to my storage account, but I am not capturing the streaming messages into my event hub. The data format that I am working with is parquet.

Below are detailed step-by-step instructions for creating an Event Hubs data connection for Azure Data Explorer and data up from the azure blob to event hub:

• Create a storage account and select the events.

• Create an Event Subscription by selecting the Filter for Event Types, choosing Endpoint Type as Event Hub, and configuring an endpoint by selecting the event type.

• Select Azure Data Explorer cluster and database. Create a cluster and database.

• Streaming ingestion must be configured on your Azure Data Explorer cluster.

Get Data

• Navigate to the Azure Data Explorer portal.

• In the portal, go to the Query section.

• Right-click on the database where you want to ingest the data and select Get data.

  

Source

• In the Get data window, select the Source tab.

• Choose Event Hubs as the data source.

Configure

• In the Configure tab, fill in the required fields:
• Subscription: The subscription ID where the Event Hub resource is located. - Event Hub Namespace: The name identifying your namespace. - Event Hub: The specific event hub you wish to connect. - Consumer Group: The consumer group defined in your event. - Data Connection Name: A unique name for your data connection. - Advanced Filters:
  • Compression: The compression type of the event hub messages payload.
  • Event System Properties: Additional event hub system properties.
  • Event Retrieval Start Date: The date from which existing Event Hubs events should be retrieved.
  • Select Next to proceed.

Inspect

• The Inspect tab provides a preview of the data. To complete the ingestion process, select Finish.

• Optionally, use commands to fetch new data or adjust settings in the Command Viewer and Schema Definition File dropdown.

  Edit Columns: Make adjustments based on your data type.

Mapping Transformations

• For certain data formats (Parquet, JSON, and Avro), apply ingest-time transformations in the Edit Columns window.