Using ccache when building inside of docker

I am working on moving the build for a C++ project into a docker image. The image will be built and pushed by a Jenkins job. Prior to docker, I made heavy use of ccache to speed up my builds on Jenkins, especially in the case of builds where very little changed. The trouble with docker is that the build now runs in an isolated environment, so I can no longer benefit from ccache. Is there a way to build inside of an ephemeral container while still taking advantage of ccache?

OK as promised.

Prereqs

1. Be on Docker 18.06 or above

2. Run in experimental mode, on client and server (at time of writing)

Setup experimental Mode

Server: I created the following systemd override file:

cat /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --experimental -H fd:// 

I then did a systemctl daemon-reload ; systemctl restart docker.service

Client side you need to set and env variable export DOCKER_BUILDKIT=1

Docker file

This is the basis of the docker file. I am building a php image, but you can build whatever you want. The key parts of the 1st line comment (needed to tell docker to parse the file in experimental mode), and the --mount=type=cache,target=/ccache/ option. This pulls in the cache folder for that stage of the build. Make sure you put it on every RUN line where you trigger a compile

# syntax = docker/dockerfile:experimental
FROM php:7.3-fpm-stretch

# Create app directory
WORKDIR /usr/src/app

# setup locales
RUN apt update && \
  apt install -y locales && \
  sed -i -e 's/# en_GB.UTF-8 UTF-8/en_GB.UTF-8 UTF-8/' /etc/locale.gen && \
  locale-gen

ENV LANGUAGE=en_GB.UTF-8
ENV LANG=en_GB.UTF-8
ENV LC_ALL=en_GB.UTF-8
ENV CCACHE_DIR=/ccache

RUN apt update ; apt install -yq \
        git \
        cloud-guest-utils \
        iproute2 \
        libxml2-dev \
        libxslt1-dev \
        libmemcached-dev \
        libgd-dev \
        libzip-dev \
        libmemcached-dev \
        ccache \
        awscli


# use ccache (make it appear in path earlier then /usr/bin/gcc etc)
RUN for p in gcc g++ cc c++; do ln -vs /usr/bin/ccache /usr/local/bin/$p;  done

# prod format
RUN --mount=type=cache,target=/ccache/ docker-php-source extract && \
    docker-php-ext-install \
    intl \
    bcmath  \
    calendar \
    exif \
    gd \
    gettext \
    mysqli \
    opcache \
    pcntl \
    pdo_mysql \
    shmop \
    soap \
    sockets \
    sysvmsg \
    sysvsem \
    sysvshm \
    xsl \
    zip \
  && \
  docker-php-source delete
RUN --mount=type=cache,target=/ccache/ ccache -s

Sample output

I am running the build with the --progress plain flag as the output in experimental is very different

#25 [16/16] RUN --mount=type=cache,target=/ccache/ ccache -s 
#25 digest: sha256:98c661a0404c71176a4bbf7d02123184524a784fabb2575d5210da088f16ee3a 
#25 name: "[16/16] RUN --mount=type=cache,target=/ccache/ ccache -s" 
#25 started: 2019-07-01 09:35:15.158199623 +0000 UTC 
#25 0.468 cache directory /ccache 
#25 0.468 primary config /ccache/ccache.conf 
#25 0.468 secondary config (readonly) /etc/ccache.conf 
#25 0.468 cache hit (direct) 2450 
#25 0.468 cache hit (preprocessed) 152 
#25 0.468 cache miss 590 
#25 0.468 cache hit rate 81.52 % 
#25 0.468 called for link 163 
#25 0.468 called for preprocessing 1011 
#25 0.468 compile failed 33 
#25 0.468 preprocessor error 3 
#25 0.468 bad compiler arguments 188 
#25 0.468 autoconf compile/link 943 #25 0.468 no input file 554 
#25 0.468 cleanups performed 0 
#25 0.468 files in cache 1288 #25 0.468 cache size 20.6 MB
#25 0.468 max cache size 5.0 GB 
#25 completed: 2019-07-01 09:35:15.732702254 +0000 UTC 
#25 duration: 574.502631ms

More readings here: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md https://docs.docker.com/engine/reference/commandline/dockerd/#description