After changing to non root & read only filesystem, tomcat giving 404

I have the .war file, it was working fine. but when run as a non-root & read-only filesystem is enabled, it starts to fail with errors like being unable to create a file or directory. So I create tmp volumes and mounted them on all paths which were giving issues.

But now there is no error in logs and Tomcat is giving 404 for all endpoints. As it is giving 404, k8s is restarting that pod.

tmp volume mounted paths

volumeMounts:
    - name: tmp1
      mountPath: /usr/local/tomcat/logs
    - name: tmp2
      mountPath: /usr/local/tomcat/temp
    - name: tmp3
      mountPath: /usr/local/tomcat/webapps/ROOT
    - name: tmp4
      mountPath: /usr/local/tomcat/conf/Catalina
    - name: tmp5
      mountPath: /usr/local/tomcat/work/Catalina

Logs without error

INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx750m
INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms256m
INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat
INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.30] using APR version [1.6.5].
INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1d  10 Sep 2019]
INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 533 ms
INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/8.5.69]
INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/usr/local/tomcat/webapps/ROOT.war]
INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/usr/local/tomcat/webapps/ROOT.war] has finished in [413] ms
INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 469 ms
INFO [Thread-4] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
INFO [Thread-4] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
INFO [Thread-4] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
INFO [Thread-4] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]

I am not sure from where should I start looking. Logs has no error.

Solution

So, I found the issue. The issue is with mountPath: /usr/local/tomcat/webapps/ROOT

tomcat wants to create a ROOT directory and I was mounting it. So Tomcat was like ROOT is already present, I am not going to do anything.

Now I changed the mount point and also small change in Dockerfile is working fine.

Mount point

Instead of mounting at /usr/local/tomcat/webapps/ROOT, mounting now at /usr/local/tomcat/webapps

- name: tmp3
  mountPath: /usr/local/tomcat/webapps

Change in Dockerfile

Before I was moving the .war file directly to /usr/local/tomcat/webapps/ROOT.war, but now we are mounting webapps so it will not be available at run time.

Copying ROOT.war to some other location and before starting tomcat, move it to webapps

ADD myservice/target/myservice.war /usr/local/tomcat/ROOT.war
COPY script.sh /script.sh

CMD ["bash", "/script.sh"]

script.sh

cp /usr/local/tomcat/ROOT.war /usr/local/tomcat/webapps/ROOT.war
catalina.sh run