Manage Scaling of Worker Role Programmatically

I'm looking into restructuring the application that I'm working on to find a way that will cut costs and give us a lot more room for scalability. In essence, the application is currently hosted on Azure as one large web app which users can log into, do some computationally expensive work on data stored in memory on the web app, and then eventually log off.

When looking into another way to scale this, one idea was to use Worker Roles. Instead of doing the processing on the web app, which currently requires us to use a fairly expensive pricing tier, we could use Service Bus to pass messages with the relevant data to a Worker Role instance, which would do this processing and send back the results.

The most cost-effective way to do this it seems, would be to create a small instance of a Worker Role for each user that logs on, which would deal exclusively with their requests (using, for example, a queue named after the user's ID) and then be destroyed when the user's session ends.

I have the code to determine when to spin up an instance, how to pass these messages back and forth and when to shut an instance down, but I'm having difficulty finding documentation for any methods or API calls that would allow me to do this easily. The closest I can find for deleting an instance is described here, but I can't find anything for creating them.

What is the best way to spin instances up and down on Azure? What alternatives are available to me? I'm also happy to hear alternative proposals on how to architect this.

The most cost-effective way to do this it seems, would be to create a small instance of a Worker Role for each user that logs on, which would deal exclusively with their requests (using, for example, a queue named after the user's ID) and then be destroyed when the user's session ends.

I would not recommend this approach. Here are my reasons:

• The number of Virtual Machine cores are limited in a subscription. Imagine a scenario that you get 1000s of users logged in into your application. Creating 1000s of Worker Role instances would not be allowed by Azure. You would need to take special permission from Microsoft to do that.
• Spinning up a VM takes time. When you create a new Worker Role deployment for your user, it is not instantaneous. Depending on the complexity of your role, it may take anywhere from 5 - 10 minutes to start a new Worker Role instance.
• It's not an effective approach. Your basic idea is to create a new Worker Role instance when a user logs in is based on the assumption that user will do some compute intensive task. What if the user doesn't want to perform this intensive task (I may be wrong here because I don't know much about your application). Then in that case, you have created a VM instance which is of no use. Again your assumption is that user will always log out. What if the user simply closes the browser? How will you detect that user has left your application and you would need to terminate the worker role instance you created for that user.
• It's not an efficient approach. The whole premise of Cloud Computing is built around shared resources. Having a VM dedicated for a user does not sound like an efficient approach.

Possible Solution

Instead of spinning of new worker role instances, may I suggest you take a look at scaling options. Basically the idea is to start with a shared pool of Worker Role instances. When a user logs in and start a task, web role writes a message in Service Bus queue which gets dequeued by a worker role instance which does the work and return the result. Set a maximum number of tasks a worker role could process. If you exceed that count, spin off a new instance of worker role. You can take a look at auto-scaling feature available in Azure Management Portal or look at some 3rd party services which can do this scaling for you.