javahadoopmrunit

Hadoop MRUnit IllegalStateException when using Hadoop-The Definitive Guide code


I am studying Hadoop from the Definitive Guide book and tried to execute this piece of code which is resulting in the error.
Example from Chapter-5. Link to Github code:
source: https://github.com/tomwhite/hadoop-book/blob/master/ch05/src/main/java/v1/MaxTemperatureMapper.java

public class MaxTemperatureMapper
    extends Mapper<LongWritable, Text, Text, IntWritable> {
@Override
public void map(LongWritable key, Text value, Context context)
        throws IOException, InterruptedException {
    String line = value.toString();
    String year = line.substring(15, 19);
    int airTemperature = Integer.parseInt(line.substring(87, 92));
    context.write(new Text(year), new IntWritable(airTemperature));
}

}

test: https://github.com/tomwhite/hadoop-book/blob/master/ch05/src/test/java/v1/MaxTemperatureMapperTest.java

public class MaxTemperatureMapperTest {
@Test
public void processesValidRecord() throws IOException, InterruptedException {
    Text value = new Text("0043011990999991950051518004+68750+023550FM-12+0382" +
            "99999V0203201N00261220001CN9999999N9-00111+99999999999");
    new MapDriver<LongWritable, Text, Text, IntWritable>()
            .withMapper(new MaxTemperatureMapper())
            .withInputValue(value)
            .withOutput(new Text("1950"), new IntWritable(-11))
            .runTest();
}

The error I'm getting is the following:

java.lang.IllegalStateException: No input was provided
at org.apache.hadoop.mrunit.MapDriverBase.preRunChecks(MapDriverBase.java:286)
at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:142)
at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:640)
at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:627)
at book.hadoopdefinitiveguide.chap5.examples.MaxTemperatureMapperTest.processesValidRecord(MaxTemperatureMapperTest.java:12)

This happens to be the first code I'm executing in hadoop and it is throwing this error. Any help is appreciated. Thanks in advance


Solution

  • As @Thomas Junblut pointed out, you need to specify a key along with the value.

    But (assuming you're using mrunit 1.0 or higher), withInputKey|Value is deprecated. You should instead be using withInput(K1 key, V1 val), where you specify bot the key and value in one method, instead of withInputValue(..).withInputKey(..). So you'd have something like this

    new MapDriver<LongWritable, Text, Text, IntWritable>()
            .withMapper(new MaxTemperatureMapper())
            .withInput(new LongWritable(), value)
            .withOutput(new Text("1950"), new IntWritable(-11))
            .runTest();
    

    The new LongWritable() is just an arbitrary key.


    EDIT

    So after further testing, it's not a problem with your code (except for the deprecation, but that isn't the cause).

    I ran the same test using the code I posted and got the exact same error. I tested with some old practice project I had. It seemed to be a problem with the classes not being built correctly. I created a whole new project, rewrote (copy-paste) the mapper class and created a new test case, saved everything, ran it, and it worked fine. Try to do this. BTW I'm on eclipse using the Eclipse Hadoop plugin to create the MR Project