Spark EMR job jackson error - java.lang.NoSuchMethodError - AnnotatedMember.getType()Lcom/fasterxml/jackson/databind/JavaType

I know we have similar questions already answered here. but for some reason none of the options are working for me.

Below is the error i'm getting:

User class threw exception: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.introspect.AnnotatedMember.getType()Lcom/fasterxml/jackson/databind/JavaType;
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector._fullSerializationType(JaxbAnnotationIntrospector.java:1550)
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector._findContentAdapter(JaxbAnnotationIntrospector.java:1502)
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.findSerializationContentConverter(JaxbAnnotationIntrospector.java:914)
at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findSerializationContentConverter(AnnotationIntrospectorPair.java:388)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.findConvertingContentSerializer(StdSerializer.java:266)
at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.createContextual(AsArraySerializerBase.java:197)

Below is the dependencies i have added. I tried removing all the jackson dependencies and added the required version of them seperately.

dependencies {
compile (group: 'org.scala-lang', name: 'scala-library', version: '2.11.8') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.3.0') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}
compile (group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.3.0') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.633') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile group: 'org.apache.httpcomponents', name: 'httpclient', version: '4.5.2'
compile group: 'com.sun.mail', name: 'javax.mail', version: '1.6.2'

testCompile group: 'junit', name: 'junit', version: '4.12'

compile group: 'net.liftweb', name: 'lift-json_2.11', version: '2.6.3'
compile group: 'net.sf.json-lib', name: 'json-lib', version: '2.4', classifier: 'jdk15'
compile group: 'net.sf.oval', name: 'oval', version: '1.31'

compile (group: 'com.bedatadriven', name: 'jackson-datatype-jts', version: '2.4') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-joda', version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: '2.11.2'
compile (group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile group: 'com.fasterxml.jackson.core', name: 'jackson-annotations', version: '2.11.2'

compile (group: 'com.fasterxml.jackson.module', name: 'jackson-module-jaxb-annotations', version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile (group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-csv', version: '2.11.2') {
     exclude group: 'com.fasterxml.jackson.core'
 }

// Some custom Jars

compile (group: 'org.springframework', name: 'spring-core', version: '2.5.6') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

}

this configuration works when i run the spark job in my local system but not in aws emr cluster.

Spark provides Jackson itself, and not the version you expect. The error is likely caused by having 2 conflicting Jackson versions in the classpath.

You have 2 options:

• force a specific Jackson version with some properties (see Classpath resolution between spark uber jar and spark-submit --jars when similar classes exist in both for instance)
• update your build definition to use the same Jackson version as provided by Spark

Spark 2.3.0 comes with Jackson 2.6.7.1 (can be checked here for instance: https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11/2.3.0).