dockerazure-devopsazure-pipelines

How to Enable Docker layer caching in Azure DevOps


I am running the below yaml script to build docker images and push into kubernetes cluster but at the same time I wanted to enable docker layer caching in the azure DevOps while building the yaml script.Could you please explain how to enable or how to add the task in azure devops to do this.

Yaml:

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  tag: 'web'
  DockerImageName: 'boiyaa/google-cloud-sdk-nodejs'


steps:
- task: Docker@2
  inputs:
    command: 'build'
    Dockerfile: '**/Dockerfile'
    tags: 'web'
  
- script: |
    echo ${GCLOUD_SERVICE_KEY_STAGING} > ${HOME}/gcp-key.json
               gcloud auth activate-service-account --key-file ${HOME}/gcp-key.json --project ${GCLOUD_PROJECT_ID_STAGING}
               gcloud container clusters get-credentials ${GCLOUD_PROJECT_CLUSTER_ID_STAGING} \
        --zone ${GCLOUD_PROJECT_CLUSTER_ZONE_STAGING} \
        --project ${GCLOUD_PROJECT_ID_STAGING}
  displayName: 'Setup-staging_credentials'


- bash: bash ./deploy/deploy-all.sh staging
  displayName: 'Deploy_script_staging'

Solution

  • Docker layer caching is not supported in Azure DevOps currently. The reason is stated as below:

    In the current design of Microsoft-hosted agents, every job is dispatched to a newly provisioned virtual machine. These virtual machines are cleaned up after the job reaches completion, not persisted and thus not reusable for subsequent jobs. The ephemeral nature of virtual machines prevents the reuse of cached Docker layers.

    However:

    1. Docker layer caching is possible using self-hosted agents. You can try creating your on-premise agents to run your build pipeline.

      You may need to disable the Job's option 'Allow scripts to access the OAuth token'. For $(System.AccessToken) is passed to docker build using a --build-arg ACCESS_TOKEN=$(System.AccessToken), and its value varies for every run, which will invalidate the cache.

    2. You can also you use Cache task and docker save/load commands to upload the saved Docker layer to Azure DevOps server and restore it on the future run. Check this thread for more information.

    3. Another workaround as described in this blog is to use --cache-from and --target in your Dockerfile.

    If the above workarounds are not satisfying, you can submit a feature request to Microsoft Develop Team. Click Suggest a Feature and choose Azure DevOps.