Environment:
I'm looking for input from anyone that has separated all of their 3rd party libraries into catalina lib (or a custom directory specified via extending catalina.properties 'common-loader') so that the application's war file can be considerably smaller.
Background: We have five webapps which share about 100 3rd party libraries that combined are about 100MB in size. While the applications may have a few unique dependencies the overlap in 3rd dependencies is over 90%.
Our code itself is only 5-15MB and te dependencies don't change frequently compared to our own code. So there's a lot of unnecessary file transfer and unpacking each day/week in our CI/CD.
I'm wanting to improve both deploy startup times as well as remote deploy times (via cargo) as deploying ~115MB remotely to a cluster seems unnecessarily time-consuming given < 15% of the code changes on a typical sprint.
My plan would be to:
Anyone have experience doing this or have any tips/gotchas to be concerned with?
I have a similar situation, which requires common libraries to be on Tomcat classpath. While, there are potentially many approaches, we follow the below process.
We kind of 'build' the Tomcat, not from source code perspective but create directories and put the JARs in specific folders (within catalina.base directory). We have a profile in pom which packages the dependencies of the libraries during building the library.
<profile>
<id>make-bundle</id>
<build>
<plugins>
<!--
1) Copies the project 'compile' dependencies to the staged_lib directory
under the 'target' directory.
-->
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies-compile</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${staged.lib.directory}</outputDirectory>
<includeScope>compile</includeScope>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
The dependency in the WAR for the libraries is 'provided'. The WAR does not pack the library JARs or their dependencies.
We then use a script to copy the generated library JARs and their dependencies in the specific folders in the Tomcat. The process is a bit long to explain, below is the script that we use, it is self-explanatory:
#!/bin/bash
#Declare colour constants
BRed='\033[1;31m'
NC='\033[0m'
# This batch file takes an existing aRCO Tomcat installation and prepares another new Tomcat installation with
# new updated platform JAR files. The build process steps for the Tomcat are as follows:
# 1) Unzip the tomcat zip file as found in the 'tczippath' to the 'tcnewrootdirpath'
# 2) Remanes the unzipped folder to apache-tomcat-xxx_aRCODevX and deteles the unnecessary directories from webapp folder
# 3) Copies the following files from the old Tomcat (as provided by 'oldtcpath') to the new Tomcat
# i) server.xml
# ii) context.xml
# iii) catalina.properties
# 4) Creates the platform_lib folder and it's sub-folers as necessary
# 5) Build the following libraries and copies the generated JAR files to 'generated_lib' and the dependencies to 'dependency_lib'
# i) platform-core
# ii) tomcat-dist
# iii) platform-connector
# iv) wfm-delegate-dist
# 6) If deployment of the built tomcat is needed to dev-share, the following additional steps are performed:
# i) The entire new Tomcat folder is zipped with 7zip
# ii) The 7zip file is uploaded to dev-share
#
# The following arguments need to be provided to the script in order as mentioned below:
# i) The full path to the Tomcat home folder of the old Tomcat
# ii) The fill path of the zipped (.zip) file of the Tomcat binary
# iii) The directory in which the above operations will be performed, can be any directory
# iv) The workspace directory path under which the individual above library projects are available
echo Give Old aRCO Tomcat Home Directory path
read oldtcpath
if test -z "$oldtcpath"
then
oldtcpath=~/SW/apache-tomcat-9.0.72_aRCODevX
fi
echo -e ${BRed}Using old aRCO Tomcat Home Directory path:${NC} $oldtcpath
echo
echo Give the full path of the Tomcat zip file
read tczippath
if test -z "$tczippath"
then
tczippath=~/Install/apache-tomcat-9.0.72.zip
fi
echo -e ${BRed}Using Tomcat Zip file Path:${NC} $tczippath
echo
echo Give the full path of the diretcory for new Tomcat - this is where the new tomcat will be built
read tcnewrootdirpath
if test -z "$tcnewrootdirpath"
then
tcnewrootdirpath=~/dev-tomcats
fi
echo -e ${BRed}Using Tomcat Zip file Path:${NC} $tcnewrootdirpath
echo
echo Give Workspace Root Directory Path [Eclipse workspace directory]
read workspaceroot
if test -z "$workspaceroot"
then
workspaceroot=~/workspaces/optimasprime
fi
echo -e ${BRed}Using Workspace Path [Eclipse Workspace path]:${NC} $workspaceroot
echo
echo eploy zipped Tomcat [y/n]
read deploy
if test -z "$deploy"
then
deploy=n
fi
echo -e ${BRed}Upload built Tomcat:${NC} $deploy
echo
# Create the complete paths of the new unzipped tomcat
# Get the name of the tomcat zip file without .zip extension, this is the name of the unzipped folder,
# add to it '_aRCODevX' for the full directory name
tczipfilenameonly=$(basename $tczippath)
newtcpath=$tcnewrootdirpath/${tczipfilenameonly%.*}_aRCODevX
echo The new tomcat path: $newtcpath
# Delete any old folder with the same name of the new tomcat path, if it exists
rm -r $newtcpath
# Extract the provided zip file to old tomcat path
7z x $tczippath -o$tcnewrootdirpath
# Rename the zipped tomcat to the new tomcat name
unzippedtcpath=$tcnewrootdirpath/${tczipfilenameonly%.*}
mv $unzippedtcpath $newtcpath
# Start duilding the new Tomcat with required files from the old tomcat and platform jars
echo Deleting the directories in new Tomcat webapp directory which are not required
rm -r $newtcpath/webapps/docs
rm -r $newtcpath/webapps/examples
rm -r $newtcpath/webapps/host-manager
rm -r $newtcpath/webapps/manager
echo Copying the catalina.properties, server.xml and context.xml from conf of old Tomcat to new Tomcat
cp $oldtcpath/conf/catalina.properties $newtcpath/conf/catalina.properties
cp $oldtcpath/conf/server.xml $newtcpath/conf/server.xml
cp $oldtcpath/conf/context.xml $newtcpath/conf/context.xml
echo Creating the platform_lib directory in new Tomcat root and generated_lib and staged_lib directories under them
mkdir $newtcpath/platform_lib
mkdir $newtcpath/platform_lib/generated_lib
mkdir $newtcpath/platform_lib/staged_lib
# Build the binaries and copy to required locations
echo Building platform-core jar gathering it\'s dependencies and copying files
cd $workspaceroot/platform-core
mvn clean package -Dmaven.test.skip=true -Pmake-bundle -U
cp -rT ./target/staged_lib $newtcpath/platform_lib/staged_lib
cp ./target/platform-core*.jar $newtcpath/platform_lib/generated_lib
echo Building platform-connector jar, gathering it\'s dependencies and copying files
cd $workspaceroot/platform-connector
mvn clean package -Dmaven.test.skip=true -Pmake-bundle -U
cp -rT ./target/staged_lib $newtcpath/platform_lib/staged_lib
cp ./target/platform-connector*.jar $newtcpath/platform_lib/generated_lib
echo Building tomcat-dist jar, gathering it\'s dependencies and copying files
cd $workspaceroot/tomcat-dist
mvn clean package -Dmaven.test.skip=true -Pmake-bundle -U
cp -rT ./target/staged_lib $newtcpath/platform_lib/staged_lib
cp ./target/tomcat-dist*.jar $newtcpath/platform_lib/generated_lib
echo Building wfm-delegate-dist jar, gathering it\'s dependencies and copying files
cd $workspaceroot/wfm-delegate-dist
mvn clean package -Dmaven.test.skip=true -Pmake-bundle -U
cp -rT ./target/staged_lib $newtcpath/platform_lib/staged_lib
cp ./target/wfm-delegate-dist*.jar $newtcpath/platform_lib/generated_lib
if [ $deploy == "y" ]
then
echo Deploying the aRCO Tomcat to Dev-share
echo Calculate the zip file name for the zip file to be uploaded
uploadzipfilename=$(basename $newtcpath).7z
echo Upload Zip File Name :: $uploadzipfilename
echo Change to the new tomcat root directory
cd $tcnewrootdirpath
echo Delete the 7zip file if it already exists from an earlier run
rm $uploadzipfilename
echo Create the zip file of the new tomcat with contents of the new tomcat
7z a $uploadzipfilename $newtcpath/
echo Upload the created 7zip file to Dev-share folder in Engg-subnet
echo Upload URL :: http://hitman3.server:8080/sw-repo/tomcat/$uploadzipfilename
curl --upload-file $uploadzipfilename http://hitman3.server:8080/sw-repo/tomcat/$uploadzipfilename -v
fi
Currently we run the above script manually but it can also be dome with Jenkins other CI/CD.
Please not, we don't make an uber-jar