Recently, I want to migrate a Storm Topology to Heron. But there are some problems with it.
In Heron topology, I can set the resources of components
and containers
, such as:
conf.setComponentRam("spout", ByteAmount.fromMegabytes(512));
conf.setComponentRam("split", ByteAmount.fromMegabytes(512));
conf.setComponentRam("count", ByteAmount.fromMegabytes(512));
conf.setContainerDiskRequested(ByteAmount.fromGigabytes(3));
conf.setContainerRamRequested(ByteAmount.fromGigabytes(3));
conf.setContainerCpuRequested(2);
But In Storm topology, I can't set these resources requirements. So when I try to running a Storm topology in Heron using RoundRobin Packing
, it will use the default resource settings
as following:
DEFAULT_DISK_PADDING_PER_CONTAINER = ByteAmount.fromGigabytes(12);
DEFAULT_CPU_PADDING_PER_CONTAINER = 1;
MIN_RAM_PER_INSTANCE = ByteAmount.fromMegabytes(192);
DEFAULT_RAM_PADDING_PER_CONTAINER = ByteAmount.fromGigabytes(2);
But the problem is: the worker node in my Heron cluster does not has so many RAM
or Disk
resources. So when submitted the Storm topology, the state of this topology will be pending
in Aurora, because of the Insufficient: disk or ram.
I am not very familiar with Storm, can I set the resource requirements of the components in the Storm topology? If not, in addition to increasing the resource capacity of the working nodes in the cluster, is there any other way to solve this problem? Any help is very grateful.
First, let me try to understand your question: You are asking about how to set cpu/memory/disk limits for each component in your topology when using the RoundRobinPacking resource packing algorithm.
Second, if I understand your question correctly. Then the answer is just the first part of your question. You can set those limits with the conf
object provided by heron framework. Some things to keep in mind are:
Third, Heron's underlying architecture is very different from Storm. I don't think Storm offers such fine-grained level of resource control. And Storm doesn't have these configs.