pythonhadoopmapreducecloudera-quickstart-vm

I am unable to execute a mapreduce job in a Hadoop cluster

$ hadoop jar /usr/lib/hadoop/hadoop-streaming-2.6.0-cdh5.13.0.jar -file mapper.py -mapper mapper.py -file reducer.py -reducer reducer.py -input /user/cloudera/test.txt -output /user/cloudera/result

I am using this command to execute MapReduce program using Mapper asmapper.py and Reducer as reducer.py

It throws an an error Not a valid JAR:/usr/lib/hadoop/hadoop-streaming-2.6.0-cdh5.13.0.jar

I am using MobaXterm and VMBox & My home directory is /user/cloudera, mapper program location /user/cloudera/mapper.py reducer location /user/cloudera/reducer.py

Solution

  • If you are using cloudera distribution for practice, the JAR will not be available in the mentioned location /usr/lib/hadoop . The hadoop-streaming JAR will be present in /usr/lib/hadoop-mapreduce/

    Run with the updated location of the JAR, it should work fine.