I'm trying to do a map reduce against a HBase table using our in-house framework which makes use of Guice 4.0
It looks like yarn might be using version 3.0 and I'm getting a type mismatch.
2016-03-25 08:19:45,784 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: com.google.inject.multibindings.MultibindingsScanner$1.binder()Lcom/shaded/google/inject/Binder;
at com.google.inject.multibindings.MultibindingsScanner$1.configure(MultibindingsScanner.java:53)
at com.google.inject.AbstractModule.configure(AbstractModule.java:59)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
at com.google.inject.AbstractModule.install(AbstractModule.java:118)
at my.packagename.client.KafkaModule.configure(KafkaModule.java:36)
at com.google.inject.AbstractModule.configure(AbstractModule.java:59)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
at com.google.inject.spi.Elements.getElements(Elements.java:101)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103)
at com.google.inject.Guice.createInjector(Guice.java:95)
at com.google.inject.Guice.createInjector(Guice.java:72)
at com.google.inject.Guice.createInjector(Guice.java:62)
at my.packagename.hbase_fix.CleanerMapper.setup(CleanerMapper.java:82)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
In the same piece of work I had a similar mismatch with fasterxml so used the maven-shade-plugin to successfully relocate but I've not been so successful with com.google.inject
<relocations>
<relocation>
<pattern>com.fasterxml</pattern>
<shadedPattern>com.shaded.fasterxml</shadedPattern>
<excludes>
<exclude>com.fasterxml.*</exclude>
</excludes>
</relocation>
<relocation>
<pattern>org.apache.commons.lang</pattern>
<shadedPattern>org.shaded.apache.commons.lang</shadedPattern>
<excludes>
<exclude>org.apache.commons.lang.*</exclude>
</excludes>
</relocation>
<relocation>
<pattern>com.google.inject.Binder</pattern>
<shadedPattern>com.shaded.google.inject.Binder</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.inject.multibindings</pattern>
<shadedPattern>com.shaded.google.inject.multibindings</shadedPattern>
</relocation>
</relocations>
Any suggestions on how to get round type mismatch in MapReduce jobs specifically with google.inject?
I don't think there is any easy solution to this. In the end it came down to forcing the mapreduce classpath to have a copy of guice4 come first so the code would run.
Not ideal but the only solution found