apache-stormheron

How to set resource requirements of components for a Storm topology running in Heron?


Recently, I want to migrate a Storm Topology to Heron. But there are some problems with it.
In Heron topology, I can set the resources of components and containers, such as:

conf.setComponentRam("spout", ByteAmount.fromMegabytes(512));
conf.setComponentRam("split", ByteAmount.fromMegabytes(512));
conf.setComponentRam("count", ByteAmount.fromMegabytes(512));
conf.setContainerDiskRequested(ByteAmount.fromGigabytes(3));
conf.setContainerRamRequested(ByteAmount.fromGigabytes(3));
conf.setContainerCpuRequested(2);

But In Storm topology, I can't set these resources requirements. So when I try to running a Storm topology in Heron using RoundRobin Packing, it will use the default resource settings as following:

DEFAULT_DISK_PADDING_PER_CONTAINER = ByteAmount.fromGigabytes(12);
DEFAULT_CPU_PADDING_PER_CONTAINER = 1;
MIN_RAM_PER_INSTANCE = ByteAmount.fromMegabytes(192);
DEFAULT_RAM_PADDING_PER_CONTAINER = ByteAmount.fromGigabytes(2);

But the problem is: the worker node in my Heron cluster does not has so many RAM or Disk resources. So when submitted the Storm topology, the state of this topology will be pending in Aurora, because of the Insufficient: disk or ram.

I am not very familiar with Storm, can I set the resource requirements of the components in the Storm topology? If not, in addition to increasing the resource capacity of the working nodes in the cluster, is there any other way to solve this problem? Any help is very grateful.


Solution

  • First, let me try to understand your question: You are asking about how to set cpu/memory/disk limits for each component in your topology when using the RoundRobinPacking resource packing algorithm.

    Second, if I understand your question correctly. Then the answer is just the first part of your question. You can set those limits with the conf object provided by heron framework. Some things to keep in mind are:

    1. You can set the container resources only. In this case, the spouts/bolts in one container will share these resources equally.
    2. You can set the resources for each type of bolt/spout. In this case, the total resources used for each container is the sum of those spouts/bolts packed into that container.
    3. You can combine the above two. In this case, the unspecified spouts/bolts will share the resources remaining.

    Third, Heron's underlying architecture is very different from Storm. I don't think Storm offers such fine-grained level of resource control. And Storm doesn't have these configs.