github-actions

Matrix strategy over entire workflow in github actions


With the matrix strategy and two jobs like those below, the first job runs a "matrix" of possible configurations in parallel, and when all of them succeed, it moves onto the second job.

jobs:
  job1:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        version: [10, 12, 14]
    steps:
      # etc

  job2:
    runs-on: ubuntu-latest
    needs: job1
    strategy:
      matrix:
        version: [10, 12, 14]
    steps:
      # etc

What would be nice is to essentially have a matrix over the whole workflow, where for each matrix value, the entire workflow of jobs is run unimpeded, in parallel. That means for a particular configuration job2 can proceed right after job1 completes for that configuration, regardless of whether job1 for another configuration is still in process or has failed.

In the following diagram, the first case is what will currently happen, and the second is the desired case:

diagram of github actions matrix

I feel like this could be "emulated" by launching the workflow several times with a different context. But how could this be done, for example from a trigger? Somehow specify on: push that 3 versions of the workflow must run simultaneously?


Solution

  • After some digging, I think "Reusable workflows" is the currently supported method.

    The solution is to have two workflow files. One that has all the jobs, and another small workflow that has the matrix and calls the other workflow.

    name: Common Workflow
    
    on:
      workflow_call:
        inputs:
          version: # the variable you can use in place of a matrix
            required: true
            type: number
    
    jobs:
      job1:
        runs-on: ubuntu-latest
        steps:
          - run: echo 'job1 version ${{ inputs.version }}'
    
      job2:
        runs-on: ubuntu-latest
        needs: job1
        steps:
          - run: echo 'job2 version ${{ inputs.version }}'
    
    name: Master Workflow
    
    on:
      push:
    
    jobs:
      version-matrix:
        strategy:
          # super important if you want to see all results, even if one fails
          # fail-fast is true by default
          fail-fast: false
          matrix:
            version: [10, 12, 14]
        uses: ./.github/workflows/common-workflow.yml # calls the one above ^
        with:
          version: ${{ matrix.version }}
        secrets: inherit
    

    Here's a real example of it running in the UI:

    Reusable Workflows UI lists jobs in alphabetical order

    One thing you will run into once you run this though, is the UI lists the subjobs in alphabetical order, and not the order they are listed in the workflow. So you lose the sense of the dependencies between the jobs. This is true in February 2023, so maybe Github will improve the UI down the road. Until then, we got around this by naming our jobs like 1. First Job, 2. Second Job so alphabetical is equivalent to the order they run in. It's hacky but it works.

    Another thing to note, especially for deployments is the fail-fast attribute. This should be set to false to make sure every workflow in the matrix gets to run to completion, even if one of them fails.