Building LLVM with direct mode distcc and ccache

I'm trying to build LLVM with both distcc (in direct mode, i.e. non-pump mode) and with a ccache on the initiating machine.

Tools in isolation

I've managed to get both of these systems to work with the LLVM build system in isolation, but not together.

Getting the LLVM build system to use ccache is easy, you just pass -DLLVM_CCACHE_BUILD=On to cmake. This works.

Getting distcc working amounts to setting up the remote build hosts and passing -DCMAKE_CXX_COMPILER_LAUNCHER="distcc" and -DCMAKE_C_COMPILER_LAUNCHER="distcc". This also works. I've done a full build spanning three machines this way.

ccache and distcc

Combining the two does not seem to work, by which I mean, after a full build to fully populate the ccache (cmake --build build), a clean (cmake --build build --target clean), and a rebuild (cmake --build build), the build is not as fast as it is when using only ccache with 100% cache hits (and I can also see build jobs being spawned on remotes, which shouldn't be necessary if all source files are cache hits).

With just ccache and a fully populated cache, I can build hundreds of files a second. But with both distcc and ccache, it's more like 5-10 a second. Much slower.

I've also tried not using -DLLVM_CCACHE_BUILD=On, but instead manually specifying the compiler launcher, e.g.: -DCMAKE_CXX_COMPILER_LAUNCHER="ccache;distcc" (and the same for the C compiler). The build succeeds, but again, I don't think it's hitting the ccache, as it's much slower than it should be.

Monitoring ccache --show-stats during a build, I do see stats changing, so something is obviously touching the cache.

And if I grep the process table, I see jobs like:

/bin/sh -c ccache distcc /usr/bin/clang++ ...

Which looks correct to me.

Has anyone managed to get this working? It would really accelerate my development turnaround.

Further details

~/.config/ccache:

cache_dir = /opt/ccache

Example cmake invocation:

cmake \
    -DLLVM_INSTALL_UTILS=On \
    -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
    -DCMAKE_CXX_COMPILER_LAUNCHER="ccache;distcc" \
    -DCMAKE_C_COMPILER_LAUNCHER="ccache;distcc" \
    -DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
    -DCMAKE_C_COMPILER=/usr/bin/clang \
    -DCMAKE_BUILD_TYPE=release \
    -DLLVM_ENABLE_ASSERTIONS=On \
    -DLLVM_ENABLE_PROJECTS="lld;clang;clang-tools-extra" \
    -G Ninja \
    ../llvm

Any ideas? Thanks.

I figured this out in the end.

The solution is to pass -DLLVM_CCACHE_BUILD=On to cmake but to also set CCACHE_PREFIX=distcc in the environment.

Using this method there is no need to pass the -DCMAKE_*_COMPILER_LAUNCHER options to cmake.

(That said, not sure why the other methods I tried didn't work)