TLDR: On Java 9/10, a web app in Tomcat has no access to JAXB even though its reference implementation is present on the class path.

Edit: No, this is not a duplicate of How to resolve java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException in Java 9 - as you can tell by the What I tried section, I already tried the proposed solutions.

The Situation

We have a web app that runs on Tomcat and depends on JAXB. During our migration to Java 9 we opted for adding the JAXB reference implementation as a regular dependency.

Everything worked when launching the app from the IDE with embedded Tomcat, but when running it on a real Tomcat instance, I get this error:

Caused by: java.lang.RuntimeException: javax.xml.bind.JAXBException:
    Implementation of JAXB-API has not been found on module path or classpath.
 - with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory]
    at [... our-code ...]
Caused by: javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.
    at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:278) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.ContextFinder.find(ContextFinder.java:421) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:721) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:662) ~[jaxb-api-2.3.0.jar:2.3.0]
    at [... our-code ...]
Caused by: java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
    at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) ~[?:?]
    at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) ~[?:?]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:499) ~[?:?]
    at javax.xml.bind.ServiceLoaderUtil.nullSafeLoadClass(ServiceLoaderUtil.java:122) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.ServiceLoaderUtil.safeLoadClass(ServiceLoaderUtil.java:155) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:276) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.ContextFinder.find(ContextFinder.java:421) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:721) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:662) ~[jaxb-api-2.3.0.jar:2.3.0]
    at [... our-code ...]

Note:

Implementation of JAXB-API has not been found on module path or classpath.

These are the relevant files in webapps/$app/WEB-INF/lib:

jaxb-api-2.3.0.jar
jaxb-core-2.3.0.jar
jaxb-impl-2.3.0.jar

What is going on here?

What I tried

Adding JARs to Tomca's CLASSPATH

Maybe it helps to add the JARs to Tomcat's class path in setenv.sh?

CLASSPATH=
    .../webapps/$app/WEB-INF/lib/jaxb-api-2.3.0.jar:
    .../webapps/$app/WEB-INF/lib/jaxb-impl-2.3.0.jar:
    .../webapps/$app/WEB-INF/lib/jaxb-core-2.3.0.jar:
    .../webapps/$app/WEB-INF/lib/javax.activation-1.2.0.jar

Nope:

Caused by: javax.xml.bind.JAXBException: ClassCastException: attempting to cast
jar:file:.../webapps/$app/WEB-INF/lib/jaxb-api-2.3.0.jar!/javax/xml/bind/JAXBContext.class to
jar:file:.../webapps/$app/WEB-INF/lib/jaxb-api-2.3.0.jar!/javax/xml/bind/JAXBContext.class.
Please make sure that you are specifying the proper ClassLoader.    
    at javax.xml.bind.ContextFinder.handleClassCastException(ContextFinder.java:157) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:300) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:286) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.ContextFinder.find(ContextFinder.java:409) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:721) ~[jaxb-api-2.3.0.jar:2.3.0]
    at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:662) ~[jaxb-api-2.3.0.jar:2.3.0]
    at de.disy.gis.webmapserver.factory.DefaultWmsRequestFactory.initializeCommandExtractor(DefaultWmsRequestFactory.java:103) ~[cadenza-gis-webmapserver-7.7-SNAPSHOT.jar:7.6]
    at de.disy.gis.webmapserver.factory.DefaultWmsRequestFactory.lambda$new$0(DefaultWmsRequestFactory.java:87) ~[cadenza-gis-webmapserver-7.7-SNAPSHOT.jar:7.6]

That's clearly the same class, so apparently it has been loaded by two class loaders. I suspect the system class loader and the app's class loader, but why would loading JAXBContext be delegated to the system class loader once but not always? It almost looks as if the delegation behavior of the app's class loader changes while the program runs.

Adding the module

I don't really want to add java.xml.bind, but I tried it anyways by adding this to catalina.sh:

JDK_JAVA_OPTIONS="$JDK_JAVA_OPTIONS --add-modules=java.xml.bind"

Doesn't work either, though:

Caused by: java.lang.ClassCastException:
java.xml.bind/com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl
cannot be cast to com.sun.xml.bind.v2.runtime.JAXBContextImpl
    at [... our-code ...]

Apart from the different class and stack trace, this is in line with what happened earlier: The class JAXBContextImpl was loaded twice, once from java.xml.bind (must have been the system class loader) and one other time (I assume by the app's loader from the JAR).

Searching for bugs

Searching Tomcat's bug database I found #62559. Could that be the same error?

Adding JAR's to Tomcat's lib

Following advice given on the Tomcat user mailing list, I added the JAXB JARs to Tomcat's CATALINA_BASE/lib directory, but got the same error as in the application's lib folder.

Analysis

First some random facts:

• if not given a class loader, JAXBContext::newInstance will use the thread's context class loader when looking for the JAXB implementation - this is the case even if you call newInstance(Class...) (one might mistakenly think it uses the provided class instances' loader)
• Tomcat builds a small class loader hierarchy to separate web applications from one another
• by not relying on the module java.xml.bind, in Java 9, JAXB classes are not loaded by the bootstrap or system class loader

So here's what happened on Java 8:

• we don't pass a class loader to JAXB (oops), so it uses the thread's context class loader
• our conjecture is that Tomcat does not explicitly set the context class loader and so it will end up being the same one that loaded Tomcat: the system class loader
• that's dandy because the system class loader sees the entire JDK and hence the JAXB implementation included therein

Java 9 enters - the piano stops playing and everybody puts down their scotch:

• we added JAXB as a regular dependency and so it is loaded by the web app's class loader
• just as on Java 8, JAXB searches the system class loader, though, and that one can't see the app's loader (only the other way around)
• JAXB fails to find the implementation and goes belly up

Solution

The solution is to make sure JAXB uses the right class loader. We know of three ways:

• call Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader()); but that's not really a good idea
• create a context resolver, but that requires JAX-WS and that feels like replacing one evil with another
• use the package-accepting variant of JAXBContext::newInstance (Javadoc from Java EE 7) that also takes a class loader and pass the correct loader, although that requires some refactoring

We used the third option and refactored towards the package-accepting variant of JAXBContext::newInstance. Menial work, but fixed the problem.

Note

User curlals provided the critical piece of information, but deleted their answer. I hope it was not because I asked for a few edits. All credit/karma should go to them! @curlals: If you restore and edit your answer, I will accept and upvote it.