I'm trying to work with both one WebJob and one Worker Role.
The WebJob will have a BlobTrigger, every time a blob is added to the container a new message will be added to an Azure Storage Queue (call it pending blobs).
Also, there will be a Worker Role which will be pooling messages from the pending blobs queue and will add the blob names to an internal blocking collection that will be processed concurrently by several tasks triggered by the Worker Role.
I've thought in this solution setting my mind in scalability and because there will be a lot of blobs arriving to the container so I don't want to have peaks of CPU consumption.
Some questions came to my mind while developing the solution:
Is there a way to check if the Azure Storage Queue has messages inside?
Queues have an ApproximateMessageCount property you can check, for queue depth (note: this isn't 100% accurate, because messages may be added/deleted while checking).
If I call the GetMessage method and the queue does not have any message the excecution will be blocked until a new message arrive?
GetMessage() is non-blocking. If there are no messages, the call returns. Note: Since you're planning on creating your own reader in a worker role, just be careful when dealing with an empty queue: If you put yourself in a tight loop and continue to blast the queue, you run the risk of exhausting the queue's 2000 transaction/second limit (and you'll probably see excessive network traffic and cpu utilization). How you implement a backoff strategy is up to you, but you'll want to incorporate a backoff of some type.