performancehadoopnetworkingmapreducetask-tracking

Hadoop CDH5 Maximum Network Performance on 10 node using 1 MapReduce TaskTracker

I'm trying to get maximum network performance with CDH5 Cloudera with 10 nodes with 1 MapReduce TaskTracker but can't seem to get the traffic to produce the traffic I need. I have 40G nic cards and currently only getting 10G at the most with is really low.

When using 1 MapReduce TaskTracker how can I produce better performance and network throughput?

Running -- 10 nodes RedHat 6.6 Current Nic driver CDH5 Kernel: 3.18.1

I've decommissioned 9 MapReduce TaskTracker nodes but still not getting good results.

Solution

  • I finally figured out that you really don't need a task tracker on data nodes for network dfsio test so I decommissioned all datanode "task trackers" and put the task tracker on the name node and now I have both write/tx and read/rx traffic now which is what I wanted for network throughput data.