How to setup Docker in Docker (DinD) on CloudBuild?

I am trying to run a script (unitest) that uses docker behind the scenes on a CI. The script works as expected on droneci but switching to CloudBuild it is not clear how to setup DinD.

For the droneci I basically use the DinD as shown here my question is, how do I translate the code to Google CloudBuild. Is it even possible?

I searched the internet for the syntax of CloudBuild wrt DinD and couldn't find something.

I managed to figure out a way to run Docker-in-Docker (DinD) in CloudBuild. To do that we need to launch a service in the background with docker-compose. Your docker-compose.yml script should look something like this.

version: '3'
services:
  dind-service:
    image: docker:<dnd-version>-dind
    privileged: true
    ports:
      - "127.0.0.1:2375:2375"
      - "127.0.0.1:2376:2376"
networks:
  default:
    external:
      name: cloudbuild

In my case, I had no problem using versions 18.03 or 18.09, later versions should also work. Secondly, it is important to attach the container to the cloudbuild network. This way the dind container will be on the same network as every container spawned during your step.

To start the service you need to add a step to your cloudbuild.yml file.

- id: start-dind
  name: docker/compose
  args: ['-f', 'docker-compose.yml', 'up', '-d', 'dind-service']

To validate that the dind service works as expected, you can just create a ping step.

- id: 'Check service is listening'
  name: gcr.io/cloud-builders/curl
  args: ["dind-service:2375"]
  waitFor: [start-dind]

Now if it works you can run your script as normal with dind in the background. What is important is to pass the DOCKER_HOST env variable so that the docker client can locate the docker engine.

 - id: my-script
   name: my-image
   script: myscript
   env:
     - 'DOCKER_HOST=tcp://dind-service:2375'

Take note, any container spawned by your script will be located in dind-service, thus if you are to do any request to it you shouldn't do it to http://localhost but instead to the http://dind-service. Moreover, if you are to use private images you will require some type of authentication before running your script. For that, you should run gcloud auth configure-docker --quiet before running your script. Make sure your docker image has gcloud installed. This creates the required authentication credentials to run your app. The credentials are saved in path relevant to the $HOME variable, so make sure your app is able to access it. You might have some problems if you use tox for example.