[SOLVED] create a new eventhub consume group to an existing eventhub and ignore old messages

create a new eventhub consume group to an existing eventhub and ignore old messages

We have an existing Azure Eventhub and 3 consume groups. the Eventhub has very high load and it keeps 7 days of messages. Now we need to create a new Eventhub consume group under this eventhub.

During testing, we noticed that the newly created consume group will start getting the very first message within the 7 day periods and process them all. due to the large volume of data for the past 7 days, the consumer of the new consume group experienced excessive loads. So what we hope to do is to ignore the old/existing data, the new consume group will just starting with the message at the time of its creation.

Wonder any option to achieve this?

Solution

Since you're moving to a new consumer group, there will be no checkpoints present for the first run. This means that readers for each partition will fall back to the default position that they're configured to use.

You can control this in your host.json using the initialOffsetOptions configuration discussed in the docs

To specify that reading should start only with events that arrive after the new Function is deployed, you'd set the option to fromEnd:

{
    "version": "2.0",
    "extensions": {
        "eventHubs": {            
            "initialOffsetOptions" : {
                "type" : "fromEnd"
            }
        }
    }
}

If instead you're looking to specify a specific point in time to start form, you'd set the option to specify that reading should start only with events that arrive after the new Function is deployed, you'd set the option to fromEnqueuedTime:

{
    "version": "2.0",
    "extensions": {
        "eventHubs": {            
            "initialOffsetOptions" : {
                "type" : "fromEnqueuedTime",
                "enqueuedTimeUtc" : "2025-01-01T20:31Z"
            }
        }
    }
}