EventHubs: Limit Transaction/Operations on dedicated storage accounts

Recently, we had one of our clients running an excessive amount of transaction API calls to the dedicated Event Hubs blob storage account. I believe they were using Event Hub Event Processor to retrieve the data. At one point For over 6 hours they ran over 300k transaction calls to ListBlobs API. These additional transactions added a $50 daily cost to our subscription.

Is there a way for us to limit/cap the transactions on Event Hub's related storage accounts ?

I have already set up an Alert rule to get notified when blob transaction reaches the threshold.

Thanks

Blob storage is used by processor types to track partition ownership and is critical infrastructure for load balancing. Your options to reduce calls:

• Increase the LoadBalancingUpdateInterval on your processor options such that load balancing cycles run less frequently. The tradeoff that you're making here is that your processor will be slower to react to other processors crashing or removed from the cluster when scaling down, leading to some events being delayed for processing.

• Use static partition assignment and remove the need for load balancing. (You would also want to set the interval to an infinite or very large duration.) The tradeoff here is that your processor will only ever work with the assigned partitions. This is generally recommended only when you're running in an orchestrated environment, like Kubernetes, where an orchestrator has responsibility for monitoring node health and restarting dead nodes.

• Create your own CheckpointStore implementation that does not use Blob storage for ownership-related activities and extend PluggableCheckpointStoreEventProcessor. The tradeoff is just the extra work needed for the implementation. The processor doesn't care where the data comes from.

Additional context

Load balancing calls per cycle

Default configuration:

• Load Balancing Interval : 30 seconds
• Ownership Duration : 2 minutes

Calls to Event Hubs:

• Query partitions (1)
• Create receiver, start reading (varies by claimed/faulted, max of owned partitions)

Calls to Storage:

• List Blobs (1)
• SetMetadata (varies by owned/claimed, max of partition count * 2)
• Upload (varies by new partitions, max of partition count)

What is load balancing doing?

Load balancing is best thought of as a naive competing consumer that runs in a giant while loop on a predictable schedule. It attempts to maintain a fair balance of the work by prioritizing its own "fair share" of partitions and will favor stability if there isn't a clear imbalance.

At a high level, each cycle will:

• Query Event Hub partitions; update state

• Renew Local Ownership

  • Based on local state
  • Each that the processor thinks it owns, call to SetMetadata. May call Upload.
  • If no longer owned, results in HTTP 412, local state updated to remove ownership (seen in the logs; indicates stolen between load balancing cycles)

• List All Ownership

  • Calls ListBlobs with metadata trait set
  • If this fails, the load balancing cycle cannot continue; local state is preserved (seen in logs; processors will fight and Event Hubs will enforce single reader)

• Calculate Ownership

  • Update state for all ownership and all unowned partitions; expired ownership is unowned
  • Determine number of active processors by looking at ownership; an active processor must own at least one partition

• Claim Ownership

  • Determine if this instance has its fair share based on count of active processors
  • If fair share is owned, do nothing; assume unowned will be claimed by a processor without its fair share
  • If unowned partitions, pick one to claim at random
  • If no unowned partitions, pick one to steal at random
  • Update storage with any change, call to SetMetadata. May call Upload.
  • If claimed by another, results in HTTP 412, local state updated to remove ownership (seen in the logs; indicates stolen between load balancing cycles)

• Determine balance stability

  • If fair share of partitions are owned and no claim was needed, assume stable

• Ensure owned partitions are being processed

  • If no current processing task, initialize and start
  • If owned partition has a completed task, capture exceptions, initialize and restart

• Calculate next cycle time

  • If greedy strategy and not stable, run immediate
  • If elapsed time was more than the load balancing interval, run immediate
  • Delay for (load balancing interval - current cycle time), then run next