javasubmithadoop-yarn

YARN REST API - submit simple job

I need help with the http request body filling. I am doing a PUT http request and I know that I am doing it right (in java).

I want to run the simple map reduce job example – calculate PI… When I use the command :

“hadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar pi 10 10”

In the hortonworks terminal it works, I try to create and run a job that does the same.

I think the problem is in the request body (in the run_mapreducer_M2.txt file). I am trying to use if with the YARN REST API.

What am I missing? Does someone have an example for using a simple MR job?

Thanks! Here is the XML file I am using for the request body:

<application-submission-context>    
	<application-id>application_1421661392788_0005</application-id>
    <application-name>pi2</application-name>	
	<queue>testqueue</queue>
    <priority>3</priority>
    <am-container-spec>      
		<environment>			
			<entry>			  
				<key>CLASSPATH</key>
				<value>/usr/hdp/2.2.0.0-2041/hadoop/conf&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/hadoop/lib/*&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/hadoop/.//*&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/./&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/*&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/.//*&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/*&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/hadoop-yarn/.//*&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/*&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/.//*&lt;CPS&gt;&lt;CPS&gt;/usr/share/java/mysql-connector-java-5.1.17.jar&lt;CPS&gt;/usr/share/java/mysql-connector-java.jar&lt;CPS&gt;/usr/hdp/current/hadoop-mapreduce-client/*&lt;CPS&gt;/usr/hdp/current/tez-client/*&lt;CPS&gt;/usr/hdp/current/tez-client/lib/*&lt;CPS&gt;/etc/tez/conf/&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/tez/*&lt;CPS&gt;/usr/hdp/2.2.0.0-2041/tez/lib/*&lt;CPS&gt;/etc/tez/conf</value>
			</entry>
		</environment>
		<commands>
			<command>hadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar pi 10 10</command>
		</commands>
    </am-container-spec>
    <unmanaged-AM>false</unmanaged-AM>
    <max-app-attempts>2</max-app-attempts>
    <resource>      
		<memory>1024</memory>
		<vCores>1</vCores>
    </resource>    
	<application-type>MAPREDUCE</application-type>
    <keep-containers-across-application-attempts>false</keep-containers-across-application-attempts>
    <application-tags>      
		<tag>Michael</tag>      
		<tag>PI example</tag>    
	</application-tags>
</application-submission-context>

Solution

  • Sorry if this is late, you've probably found the answer already but you want to use a POST http request instead of PUT.