Job "waiting for pending jobs" is skipped

Recently GitHub Actions workflow started behaving strangely.

I have this workflow:

odoo job supposed to be run, but it just gets skipped, even though it says it is waiting for pending jobs.

Interestingly, if I re-rerun workflow (from GitHub UI), it then triggers expected job correctly and workflow completes properly.

This particular run is triggered on tag push. Workflow looks like this:

---
# Build + Test + Deploy
name: BTD
on:
  push:

jobs:
  # This is only relevant if this workflow was triggered by tag push.
  prev_git_tag_info:
    runs-on: [self-hosted, pool]
    outputs:
      tag: ${{ steps.tag-info.outputs.tag }}
      rel_tag: ${{ steps.tag-info.outputs.rel_tag }}
      explanation: ${{ steps.tag-info.outputs.explanation }}
    steps:
      - name: Checkout monodoo
        uses: actions/checkout@v4
        with:
          # NOTE. This is needed for listdir, to make sure all related
          # commits between branches are found!
          fetch-depth: 0
      - name: get previous tag info
        id: tag-info
        uses: ./.github/actions/git_rel_tag
        with:
          tag: ${{ github.ref_name }}
          # Previous tag is one position to the right relative to provided tag.
          pos: "1"

  versions:
    runs-on: [self-hosted, pool]
    needs: [prev_git_tag_info]
    outputs:
      names_comma: ${{ steps.versions-getter.outputs.names_comma }}
      names_json: ${{ steps.versions-getter.outputs.names_json }}
    steps:
      - name: Checkout monodoo
        uses: actions/checkout@v4
        with:
          # NOTE. This is needed for listdir, to make sure all related
          # commits between branches are found!
          fetch-depth: 0
      - name: List Versions
        id: versions-getter
        uses: ./.github/actions/listdir
        with:
          paths: src/versions
          # There is one exception, where we include all paths, even if they
          # are not changed. Its when we have single tag that was just pushed.
          # This tag is not comparable with other tags, so we won't know what
          # changed. Thus we include all changes.
          only_changed: ${{ needs.prev_git_tag_info.outputs.explanation != 'single' }}
          dependency_paths: src/common
          # NOTE. These will only be truthy if tag was pushed.
          force_before_ref: ${{ needs.prev_git_tag_info.outputs.rel_tag }}
          force_after_ref: ${{ needs.prev_git_tag_info.outputs.tag }}

  odoo:  # This is job that started being skipped for unknown reason
    needs: [versions]
    if: ${{ needs.versions.outputs.names_comma != '' }}
    strategy:
      fail-fast: false
      matrix:
        version: "${{ fromJSON( needs.versions.outputs.names_json ) }}"
    uses: ./.github/workflows/_odoo.yml
    with:
      version: ${{ matrix.version }}
    secrets: inherit # pragma: allowlist secret

  # This is a dummy job, to just wait for all tests to pass. This way
  # we can specify single required status check instead of needing to
  # specify all projects (which is kind of dynamic).
  test_aggregate_success:
    runs-on: [self-hosted, pool]
    needs: odoo
    steps:
      - name: Pass
        run: echo "Tests Passed"
  test_aggregate_fail:
    runs-on: [self-hosted, pool]
    needs: odoo
    if: |
      always() &&
      contains(needs.*.result, 'failure') ||
      contains(needs.*.result, 'cancelled')
    steps:
      - uses: actions/github-script@v7
        with:
          script: |
            core.setFailed('Previous jobs failed or were cancelled. Forcing failure')

  tag_release:
    runs-on: [self-hosted, pool]
    needs: [prev_git_tag_info, versions, test_aggregate_success]
    # v1 tag is major tag aliasing latest semver tag.
    if: |
      startsWith(github.ref, 'refs/tags') &&
      needs.versions.outputs.names_comma != '' &&
      github.ref != 'refs/tags/v1'
    strategy:
      matrix:
        version: "${{ fromJSON( needs.versions.outputs.names_json ) }}"
    steps:
      - name: Checkout monodoo
        uses: actions/checkout@v4
        with:
          # To see all git tags.
          fetch-depth: 0
      - name: Get projects to tag
        id: projects
        uses: ./.github/actions/project_names
        with:
          version: ${{ matrix.version }}
          only_changed: ${{ needs.prev_git_tag_info.outputs.explanation != 'single' }}
          force_before_ref: ${{ needs.prev_git_tag_info.outputs.rel_tag }}
          force_after_ref: ${{ needs.prev_git_tag_info.outputs.tag }}
      - name: Get tag version
        if: steps.projects.outputs.names_comma != ''
        id: tag_version
        uses: ./.github/actions/format_version
        with:
          # We must provide version as prefix, because with multiple odoo
          # versions and same project name, we need to distinguish release
          # Docker tags from each other.
          major: ${{ matrix.version }}
          # ref_name here is git tag
          tag: ${{ github.ref_name }}
      - name: print version
        run: echo "${{ steps.tag_version.outputs.version }}"
      - name: Tag release
        if: steps.projects.outputs.names_comma != ''
        uses: ./.github/actions/tagit_ref
        with:
          version: ${{ matrix.version }}
          projects: ${{ steps.projects.outputs.names_comma }}
          dest_tags:
            ${{ steps.tag_version.outputs.version }},${{ matrix.version }}-stable
          DOCKER_REGISTRY: ${{ vars.DOCKER_REGISTRY }}
          DOCKER_REPO_ROOT: ${{ vars.DOCKER_REPO_ROOT }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  deploy_prod:
    needs: tag_release
    concurrency: deploying_to_prod-16-demo
    if: |
      always() &&
      startsWith(github.ref, 'refs/tags') &&
      github.ref != 'refs/tags/v1' &&
      !contains(needs.*.result, 'failure') &&
      !contains(needs.*.result, 'cancelled')
    uses: ./.github/workflows/_playbook.yml
    with:
      name: deploy-prod.yml
      # Deploying to demo environment to see if deployment would not
      # fail before deploying to real production environment
      target: prod-16-demo
    secrets: inherit # pragma: allowlist secret

Here is _odoo.yml:

---
name: Build and Test Odoo
on:
  workflow_call:
    inputs:
      version:
        required: true
        type: string
        description: Odoo major version (e.g. 16.0)

jobs:
  projects:
    runs-on: [self-hosted, pool]
    outputs:
      names_comma: ${{ steps.projects.outputs.names_comma }}
      names_space: ${{ steps.projects.outputs.names_space }}
      names_json: ${{ steps.projects.outputs.names_json }}
    steps:
      - name: Checkout monodoo
        uses: actions/checkout@v4
        with:
          # NOTE. here we need this to be able to see all tags if tag
          # push triggered this!
          fetch-depth: 0
      - name: Get previous tag info
        id: tag-info
        uses: ./.github/actions/git_rel_tag
        with:
          tag: ${{ github.ref_name }}
          # Previous tag is one position to the right relative to provided tag.
          pos: "1"
      - name: List projects
        id: projects
        uses: ./.github/actions/project_names
        with:
          version: ${{ inputs.version }}
          # TODO: we should create extra action that wraps project names
          # call with tag handling, because now we do this in multiple
          # places!
          only_changed: ${{ steps.tag-info.outputs.explanation != 'single' }}
          # NOTE. These will only be truthy if tag was pushed.
          force_before_ref: ${{ steps.tag-info.outputs.rel_tag }}
          force_after_ref: ${{ steps.tag-info.outputs.tag }}

  monodoo_tag_getter:
    runs-on: [self-hosted, pool]
    outputs:
      name: ${{ steps.tag-getter.outputs.name }}
    steps:
      - name: Checkout monodoo
        uses: actions/checkout@v4
      - name: Get Monodoo reference Docker Tag
        id: tag-getter
        uses: ./.github/actions/ref_tag_getter
        with:
          version: ${{ inputs.version }}
          paths_pattern: src/common,src/versions/$VERSION/monodoo,src/versions/$VERSION/extra
          shorten: true

  project_tag_getter:
    runs-on: [self-hosted, pool]
    outputs:
      name: ${{ steps.tag-getter.outputs.name }}
    steps:
      - name: Checkout monodoo
        uses: actions/checkout@v4
      - name: Get Odoo projects reference Docker Tag
        id: tag-getter
        uses: ./.github/actions/ref_tag_getter
        with:
          version: ${{ inputs.version }}
          # projects ref depend on monodoo and extra parents, so we must
          # include those into generated reference.
          # https://github.com/skorokithakis/shortuuid/blob/master/shortuuid/main.py
          paths_pattern: src/common,src/versions/$VERSION/monodoo,src/versions/$VERSION/extra,src/versions/$VERSION/projects
          # We must shorten generated ref, because it can go over maximum
          # docker tag length (120 symbols).
          shorten: true

  monodoo_build_exists:
    runs-on: [self-hosted, pool]
    needs: [monodoo_tag_getter]
    outputs:
      exists: "${{ steps.existence.outputs.exists }}"
    steps:
      - name: Login to docker registry
        uses: docker/login-action@v3
        with:
          registry: ${{ vars.DOCKER_REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Checkout monodoo
        uses: actions/checkout@v4
      - name: Check if images exist
        id: existence
        uses: ./.github/actions/image_existence
        with:
          repo: ${{ vars.DOCKER_REPO_ROOT }}
          names: monodoo
          tag: ${{ needs.monodoo_tag_getter.outputs.name }}

  project_build_exists:
    runs-on: [self-hosted, pool]
    needs: [projects, project_tag_getter]
    outputs:
      exists: "${{ steps.existence.outputs.exists }}"
    steps:
      - name: Login to docker registry
        uses: docker/login-action@v3
        with:
          registry: ${{ vars.DOCKER_REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Checkout monodoo
        uses: actions/checkout@v4
      - name: Check if images exist
        id: existence
        uses: ./.github/actions/image_existence
        with:
          repo: ${{ vars.DOCKER_REPO_ROOT }}
          names: ${{ needs.projects.outputs.names_comma }}
          tag: ${{ needs.project_tag_getter.outputs.name }}

  # TODO: monodoo image should be built only when its src code changed,
  # but then specific reference tag should be tagged as main tag (on tag_trunk
  # job), like 16.0 etc, so we could have persisting latest monodoo image,
  # not just reference image that is pruned periodically and will need
  # to be rebuilt.
  build_monodoo:
    runs-on: [self-hosted, pool]
    needs: [monodoo_build_exists, monodoo_tag_getter]
    steps:
      - name: Checkout monodoo
        uses: actions/checkout@v4
      - name: Prepare bake data
        if: needs.monodoo_build_exists.outputs.exists != 'true'
        uses: ./.github/actions/bake_data
        id: bake-data
        with:
          target: monodoo
          version: ${{ inputs.version }}
      - name: Build Monodoo Image
        if: needs.monodoo_build_exists.outputs.exists != 'true'
        uses: ./.github/actions/build_odoo
        with:
          version: ${{ inputs.version }}
          name: monodoo
          tag: ${{ needs.monodoo_tag_getter.outputs.name }}
          registry: ${{ vars.DOCKER_REGISTRY }}
          file: ${{ steps.bake-data.outputs.file }}
          build_contexts: |
            ${{ steps.bake-data.outputs.build_contexts }}
          SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  build_project:
    runs-on: [self-hosted, pool]
    needs:
      - projects
      - project_build_exists
      - build_monodoo
      - monodoo_tag_getter
      - project_tag_getter
    # Build it only if there is at least one project to test on.
    if: ${{ needs.projects.outputs.names_comma != '' }}
    strategy:
      matrix:
        name: "${{ fromJSON( needs.projects.outputs.names_json ) }}"
    steps:
      - name: Checkout monodoo
        uses: actions/checkout@v4
      - name: Prepare bake data
        if: needs.project_build_exists.outputs.exists != 'true'
        uses: ./.github/actions/bake_data
        id: bake-data
        with:
          target: ${{ matrix.name }}
          version: ${{ inputs.version }}
          # monodoo as a base is passed using `build_arg`, so we must
          # exclude it from build_contexts, because it won't be found
          # when trying to build an image (image here is built using
          # build, not bake command!)
          rem_key_paths: target,${{ matrix.name }},contexts,monodoo
      - name: Build project image
        if: needs.project_build_exists.outputs.exists != 'true'
        uses: ./.github/actions/build_odoo
        with:
          version: ${{ inputs.version }}
          name: ${{ matrix.name }}
          tag: ${{ needs.project_tag_getter.outputs.name }}
          registry: ${{ vars.DOCKER_REGISTRY }}
          checkout_submodules: false
          file: ${{ steps.bake-data.outputs.file }}
          build_contexts: |
            ${{ steps.bake-data.outputs.build_contexts }}
          build_args: |
            BASE=${{ vars.DOCKER_REPO_ROOT }}/monodoo:${{ needs.monodoo_tag_getter.outputs.name }}
          SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  test_odoo:
    runs-on: [self-hosted, pool]
    needs: [projects, project_tag_getter, build_project]
    # Run it only if there is at least one project to test on.
    if: ${{ needs.projects.outputs.names_comma != '' }}
    strategy:
      # For now we want other project tests to run even if others fail.
      fail-fast: false
      matrix:
        name: "${{ fromJSON( needs.projects.outputs.names_json ) }}"
    steps:
      - name: Create private key to access private repos.
        uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
      - name: Checkout monodoo
        uses: actions/checkout@v4
      - name: Login to docker registry
        uses: docker/login-action@v3
        with:
          registry: ${{ vars.DOCKER_REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Get Init Odoo modules
        uses: ./.github/actions/init_odoo_modules
        id: modules-getter
        with:
          path: src/versions/${{ inputs.version }}/projects/${{ matrix.name }}/odoo.yml
          base_path_substitutes: "/opt/odoo:src/versions/${{ inputs.version }}/monodoo"
      - name: Enable problemMatcher for Odoo
        run: echo "::add-matcher::.github/odoo-matcher.json"
      - name: Run Odoo Tests
        uses: ./.github/actions/test_odoo
        with:
          version: ${{ inputs.version }}
          project: ${{ matrix.name }}
          tag: ${{ needs.project_tag_getter.outputs.name }}
          init_paths: /opt/odoo/projects/custom
          init_modules: ${{ steps.modules-getter.outputs.init_modules }}
          skip_standard: "1"
      # Sanity checks to make sure maraplus setup does not fail.
      - name: Run Maraplus in stage mode
        uses: ./.github/actions/maraplus
        with:
          version: ${{ inputs.version }}
          project: ${{ matrix.name }}
          tag: ${{ needs.project_tag_getter.outputs.name }}
          mode: stage
      - name: Run Maraplus in prod mode
        uses: ./.github/actions/maraplus
        with:
          version: ${{ inputs.version }}
          project: ${{ matrix.name }}
          tag: ${{ needs.project_tag_getter.outputs.name }}
          mode: prod
      - name: Disable problemMatcher for Odoo
        run: echo "::remove-matcher owner=odoo::"

The root cause was that special tag v1 was introduced (via nowactions/update-majorver@v1 action). This tag is automatically created to represent latest git tag. This interferes with tags comparison during tag push, determining if there were any new changes. And because this tag points to same commit as latest tag, if it is included in comparison, it shows that there were no difference between latest and previous tag.

Fix was to exclude this special tag from comparison, to only compare tags that point to different commits explicitly:

jobs:
  # This is only relevant if this workflow was triggered by tag push.
  prev_git_tag_info:
    runs-on: [self-hosted, pool]
    outputs:
      tag: ${{ steps.tag-info.outputs.tag }}
      rel_tag: ${{ steps.tag-info.outputs.rel_tag }}
      explanation: ${{ steps.tag-info.outputs.explanation }}
    steps:
      - name: Checkout monodoo
        uses: actions/checkout@v4
        with:
          # NOTE. This is needed for listdir, to make sure all related
          # commits between branches are found!
          fetch-depth: 0
      - name: get previous tag info
        id: tag-info
        uses: ./.github/actions/git_rel_tag
        with:
          tag: ${{ github.ref_name }}
          # Previous tag is one position to the right relative to provided tag.
          pos: "1"
          excluded_tags: "v1"