We have the following Dockerfile (I have removed the unecessary parts) which we have used for well over a year now and suddenly, when it gets to the gpg --batch --verify... step the build hangs during CodeBuild and ultimately times out. I have been researching for the past few days, but I cannot find anything related.
FROM amazoncorretto:11-alpine-jdk
ENV PATH /usr/local/tomee/bin:$PATH
RUN mkdir -p /usr/local/tomee
ENV TZ America/New_York
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ /etc/timezone
WORKDIR /usr/local/tomee
# add the things we need to build the image
RUN apk update \
&& apk add sudo \
&& apk add tar \
&& apk add gpg \
&& apk add curl \
&& apk add gpg-agent \
&& apk add bash
# add the users and sudo for Lacework
RUN adduser -S tomee
RUN addgroup tomee \
&& addgroup tomee tomee
RUN addgroup sudo \
&& addgroup tomee sudo
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
# download and import the GPG keys
RUN set -x \
&& curl -fsSL 'https://www.apache.org/dist/tomee/KEYS' -o GPG_KEYS | awk -F ' = ' '$1 ~ /^ +Key fingerprint$/ { gsub(" ", "", $2); print $2 }' | sort -u \
&& gpg --import GPG_KEYS
# verify keys
RUN set -xe \
&& for key in $GPG_KEYS; do \
gpg --batch --keyserver hkp://keyserver.ubuntu.com --recv-keys "$key" || \
gpg --batch --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys "$key" ; \
done
# TOMEE variables
# Apache changes the version from time to time and removes the old version.
# When they do that, you have to go to https://dist.apache.org/repos/dist/release/tomee/ to what see version is available
# then match that here
ENV TOMEE_VER 8.0.15
ENV TOMEE_BUILD webprofile
# set up Apache/TOMEE
RUN set -x \
&& curl -fSL https://dist.apache.org/repos/dist/release/tomee/tomee-${TOMEE_VER}/apache-tomee-${TOMEE_VER}-${TOMEE_BUILD}.tar.gz.asc -o tomee.tar.gz.asc \
&& curl -fSL https://dist.apache.org/repos/dist/release/tomee/tomee-${TOMEE_VER}/apache-tomee-${TOMEE_VER}-${TOMEE_BUILD}.tar.gz -o tomee.tar.gz \
&& echo "verifying gpg signature" \
&& gpg --list-keys \
&& gpg --batch --verify tomee.tar.gz.asc tomee.tar.gz \
&& tar -zxf tomee.tar.gz \
&& mv apache-tomee-${TOMEE_BUILD}-${TOMEE_VER}/* /usr/local/tomee \
&& rm -Rf apache-tomee-${TOMEE_BUILD}-${TOMEE_VER} \
&& rm -Rf /usr/local/tomee/webapps/docs \
&& rm bin/*.bat \
&& rm tomee.tar.gz* \
&& chown -R tomee:tomee /usr/local/tomee
# put everything in the right place with the correct permissions
COPY $PWD/server.xml /usr/local/tomee/conf/
RUN chown -R tomee:tomee /usr/local/tomee/conf
COPY $PWD/target/application/META-INF/ /usr/local/tomee/webapps/ROOT/META-INF/
COPY $PWD/target/application/WEB-INF/ /usr/local/tomee/webapps/ROOT/WEB-INF/
COPY $PWD/target/application.war /usr/local/tomee/webapps/ROOT.war
RUN chown -R tomee:tomee /usr/local/tomee/webapps/
USER tomee
EXPOSE 8080
CMD ["catalina.sh", "run"]
To replicate this, I set up an EC2 instance and installed Docker on the instance. Here is where the build hangs:
+ curl -fSL https://dist.apache.org/repos/dist/release/tomee/tomee-8.0.15/apache-tomee-8.0.15-webprofile.tar.gz.asc -o tomee.tar.gz.asc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 833 100 833 0 0 17455 0 --:--:-- --:--:-- --:--:-- 17723
+ curl -fSL https://dist.apache.org/repos/dist/release/tomee/tomee-8.0.15/apache-tomee-8.0.15-webprofile.tar.gz -o tomee.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 47.4M 100 47.4M 0 0 118M 0 --:--:-- --:--:-- --:--:-- 118M
+ echo 'verifying gpg signature'
verifying gpg signature
+ gpg --batch --verify tomee.tar.gz.asc tomee.tar.gz
gpg: Signature made Mon May 8 12:36:19 2023 UTC
gpg: using RSA key B83D15E72253ED1104EB4FBBDAB472F0E5B8A431
If I comment out the batch verify, the build completes successfully.
REPOSITORY TAG IMAGE ID CREATED SIZE
gpg-test latest 1aead64a5120 12 seconds ago 350MB
amazoncorretto 11-alpine-jdk 60ba21c1871e 2 weeks ago 274MB
Now I can run the image. When I bash into the image I can see the GPG_KEYS file so I run the import:
gpg --import GPG_KEYS
That works properly, so I get the Apache-Tomee downloads:
curl -fSL https://dist.apache.org/repos/dist/release/tomee/tomee-8.0.15/apache-tomee-8.0.15-webprofile.tar.gz.asc -o tomee.tar.gz.asc
curl -fSL https://dist.apache.org/repos/dist/release/tomee/tomee-8.0.15/apache-tomee-8.0.15-webprofile.tar.gz -o tomee.tar.gz
So the container now has the files and I run the batch verify command manually and it completes successfully:
5ab9673fdbe2:/usr/local/tomee$ gpg --batch --verify tomee.tar.gz.asc tomee.tar.gz
gpg: Signature made Mon May 8 12:36:19 2023 UTC
gpg: using RSA key B83D15E72253ED1104EB4FBBDAB472F0E5B8A431
gpg: Good signature from "Richard Zowalla (Code Signing Key) <rzo1@apache.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: B83D 15E7 2253 ED11 04EB 4FBB DAB4 72F0 E5B8 A431
Why is gpg --batch --verify tomee.tar.gz.asc tomee.tar.gz failing during the building of the image?
As a side note: one of our developers is running Docker Desktop and his build process hangs at the exact same spot. Others who are building the images with Lima (https://itnext.io/replace-docker-desktop-with-lima-88ec6f9d6a19) do not encounter the problem so it leads me to believe that something in the Docker eco-system is causing an issue.
Have I missed anything obvious?
UPDATE
I added gpg --list-keys to the Dockerfile immediately above the batch verify line and it hangs at the list-keys command. All of the other GPG commands work prior to that.
I added the -v to the list keys command and I am getting the following:
verifying gpg signature
+ gpg --list-keys -v
gpg: enabled compatibility flags:
gpg: using pgp trust model
gpg: no running keyboxd - starting '/usr/libexec/keyboxd'
gpg: waiting for the keyboxd to come up ... (5s)
UPDATE
I have solved the keyboxd issue by adding && export GNUPGHOME="$(mktemp -d)" \ prior to any GPG command.
I believe I have isolated the issue to this line:
&& gpg --import GPG_KEYS
The import occurs, but when I try to loop through the keys there is no $GPG_KEYS environment variable, so there is no public key available. You can see here that the OP actually spells out the key listing in the Dockerfile, but that seems inefficient to me. Running TomEE API in Docker container successful, but reverse proxy returns a 404
YET ANOTHER UPDATE I have been fiddling with this quite a bit today and finally was able to cleanly get the GPG keys exported to an ENV variable as you can see here:
+ export 'GPG_KEYS=/tmp/tmp.hoJlmJ/pubring.kbx
---------------------------
pub dsa1024 2006-01-05 [SC]
9056B710F1E332780DE7AF34CBAEBE39A46C4CA1
uid [ unknown] Some Dude <somedude@apache.org>
sub elg2048 2006-01-05 [E]
pub rsa2048 2014-01-21 [SC]
F067B8140F5DD80E1D3B5D92318242FE9A0B1183
uid [ unknown] Another Dude <anotherdudek@apache.org>
sub rsa2048 2014-01-21 [E]
...and so on
Running printenv inside the image build shows the keys
+ printenv
HOSTNAME=756af5d791b0
SHLVL=1
HOME=/root
GPG_KEYS=/tmp/tmp.hoJlmJ/pubring.kbx
---------------------------
pub dsa1024 2006-01-05 [SC]
9056B710F1E332780DE7AF34CBAEBE39A46C4CA1
uid [ unknown] Some Dude <somedude@apache.org>
sub elg2048 2006-01-05 [E]
pub rsa2048 2014-01-21 [SC]
F067B8140F5DD80E1D3B5D92318242FE9A0B1183
uid [ unknown] Another Dude <anotherdudek@apache.org>
sub rsa2048 2014-01-21 [E]
...and so on
When I run the next command manually, it works. But running it in the Dockerfile fails (we should see output)
Step 13/21 : RUN set -xe && for key in $GPG_KEYS; do gpg -v --batch --keyserver hkp://keyserver.ubuntu.com --recv-keys "$key" ; done
None of the output I would expect from the --keyserver command is present.
We continued to research and set up a support request (unanswered) with AWS for why GPG started failing in Dockerfiles in CodeBuild.
All of the testing mentioned led to one conclusion, GPG calls worked properly from the command line on an EC2 instance and within a Docker container on that instance. Given that, we moved all of the GPG commands to a bash file, which is then called in the Dockerfile:
gpg-verify.sh
#!/bin/bash
# download and import the GPG keys
export GNUPGHOME="$(mktemp -d)"
curl -fsSL 'https://www.apache.org/dist/tomee/KEYS' -o GPG_KEYS | awk -F ' = ' '$1 ~ /^ +Key fingerprint$/ { gsub(" ", "", $2); print $2 }' | sort -u
gpg --batch --import GPG_KEYS
export GPG_KEYS="$(gpg --list-keys)"
printenv
# verify keys
for key in $GPG_KEYS; do
gpg -v --batch --keyserver hkp://keyserver.ubuntu.com --recv-keys "$key" ;
done
# set up Apache/TOMEE
curl -fSL https://dist.apache.org/repos/dist/release/tomee/tomee-${TOMEE_VER}/apache-tomee-${TOMEE_VER}-${TOMEE_BUILD}.tar.gz.asc -o tomee.tar.gz.asc
curl -fSL https://dist.apache.org/repos/dist/release/tomee/tomee-${TOMEE_VER}/apache-tomee-${TOMEE_VER}-${TOMEE_BUILD}.tar.gz -o tomee.tar.gz
gpg --list-keys -v
gpg -v --batch --verify tomee.tar.gz.asc tomee.tar.gz
Updated Dockerfile:
FROM amazoncorretto:11-alpine-jdk
ENV PATH /usr/local/tomee/bin:$PATH
RUN mkdir -p /usr/local/tomee
ENV TZ America/New_York
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ /etc/timezone
WORKDIR /usr/local/tomee
# add the things we need to build the image
RUN apk update \
&& apk add sudo \
&& apk add tar \
&& apk add gnupg \
&& apk add curl \
&& apk add gpg-agent \
&& apk add bash \
&& apk add ca-certificates
# add the users and sudo for Lacework
RUN adduser -S tomee
RUN addgroup tomee \
&& addgroup tomee tomee
RUN addgroup sudo \
&& addgroup tomee sudo
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
# TOMEE variables
# Apache changes the version from time to time and removes the old version.
# When they do that, you have to go to https://dist.apache.org/repos/dist/release/tomee/ to what see version is available
# then match that here
ENV TOMEE_VER 8.0.15
ENV TOMEE_BUILD webprofile
# GPG Verification / TOMEE Download
COPY $PWD/gpg-verify.sh .
RUN ./gpg-verify.sh
RUN set -x \
&& tar -zxf tomee.tar.gz \
&& mv apache-tomee-${TOMEE_BUILD}-${TOMEE_VER}/* /usr/local/tomee \
&& rm -Rf apache-tomee-${TOMEE_BUILD}-${TOMEE_VER} \
&& rm -Rf /usr/local/tomee/webapps/docs \
&& rm bin/*.bat \
&& rm tomee.tar.gz* \
&& chown -R tomee:tomee /usr/local/tomee
# put everything in the right place with the correct permissions
#COPY $PWD/server.xml /usr/local/tomee/conf/
RUN chown -R tomee:tomee /usr/local/tomee/conf
#COPY $PWD/target/application/META-INF/ /usr/local/tomee/webapps/ROOT/META-INF/
#COPY $PWD/target/application/WEB-INF/ /usr/local/tomee/webapps/ROOT/WEB-INF/
#COPY $PWD/target/application.war /usr/local/tomee/webapps/ROOT.war
RUN chown -R tomee:tomee /usr/local/tomee/webapps/
USER tomee
EXPOSE 8080
CMD ["catalina.sh", "run"]
This, of course, seems quite counter-intuitive as the bash commands are the same commands that were being run within the Dockerfile. However, this works as we expect.
P.S. We have asked AWS about changes that would've caused the issues with GPG and will update this when a response is provided.