Bazel: Mixing a Linux remote execution platform with a Mac OS local platform

Goal

I am using Bazel to build a multiplatform C++ client (iOS, OSX, Android, Windows).

iOS and OSX are built locally on my Mac (out of necessity). Android and Windows are built inside a Docker container.

At the end of the build I have a Bazel rule that takes each cc_binary rule for each platform and puts them in a .zip.

I'd like to utilize Bazel's remote execution API to build some of my binaries in the container and others locally and then reference a shared cache to collate them together -- all with one bazel build command.

Bazel support

Bazel says that these types of multiplatform builds -- where the host (OSX x64), execution (Linux x64), and target platforms (many) are all different -- are possible.

See https://docs.bazel.build/versions/master/platforms.html

My experience

However, I hit these below issues:

https://github.com/bazelbuild/bazel/issues/5397 (where docker-sandbox is a correct proxy for remote builds.)

https://github.com/bazelbuild/bazel/issues/5309

These issues seem to say that local targets for one platform (e.g., OSX) cannot be built alongside remote targets on another platform (e.g. Linux).

Question

I was wondering:

(1) Is what I am trying to do fundamentally at odds with Bazel's design? If so, what is meant by Bazel being multiplatform?

(2) Is there a workaround I can employ that maintains hermiticity and stays within the Bazel build system? It could be possible to mount a Docker volume and then write a script that combines the Docker cache with my local cache, but it seems like Bazel was built to handle my use case. Am I missing something here?

Solution

(1) Is what I am trying to do fundamentally at odds with Bazel's design?

In theory no, in practice yes. Bazel provides functionality which allows users to support your use-case, but it is not implemented by default.

Specifically, as described in the linked Bazel issues: Bazel rules currently make assumptions about the relationship between the host and target platforms which don't hold in your case, e.g. it will auto-detect the JDK files on your host (macOS) and then default to using these JDK files across all Java actions - regardless of target platform.

If so, what is meant by Bazel being multiplatform?

In practice, it means that you can run bazel build ... on multiple platforms and expect that Bazel will transform your inputs into outputs compatible for the current platform.

(2) Is there a workaround I can employ that maintains hermiticity and stays within the Bazel build system?

Yes, you can run bazel build ... from within a Windows VM or Docker container. This was the workaround that the Bazel team recommended when I asked this question.

Relevant advanced Bazel features:

If you want to build for multiple target platforms with one Bazel invocation, have a look at Bazel user-defined transitions (this would allow you to build the same rule for multiple platforms, e.g. iOS and macOS at once, but require you to write your own rules).
If you don't want to run bazel build from within containers/VMs, you can write your own C++ toolchain. At its core, Bazel gives each action a sandbox with all dependent files and guarantees that it will execute a specific command. In a custom C++ toolchain, you could tell Bazel to call a script instead of clang, which takes the command + the files and executes them from within a VM or container. This is likely a lot of work, but is definitely possible.