javahadoopmapreducereducerswritable

How do you handle different value types in reducer

I am writing a mapreduce program which has 2 mappers and 1 reducer, I implemented custom Writable Datatypes for each mapper. The Datatype is more or less just a container where the fields are Text / Intwritable values.

So Mapper 1 outputs id(Text), M1Writable (my writable with 3 fields)

Mapper 2 outputs id(Text), M2Writable (my writable with 2 fields)

The reducer will get an iterable values

how will this work since the two types of writable are different?

How can I determine which mapper it came from?

Thanks I know this is a basic question, but I was having issues finding an answer.

Solution

  • Mappers in MapReduce always output the same type.

    Unless M1Writable and M2Writable have a common parent class (e.g., MWritable), which is the common output type of all mappers, you cannot output different types among your mappers.

    To know which mapper the output came from, you need to record the information in your custom MWritable objects.