gitazure-pipelines

Azure pipeline checkout with submodules fails

I have a pipeline on Azure that is supposed to checkout repository RA from project A, which has a submodule from repository RB on project B.

I define both in the resources section:

resources:
  repositories:
    - repository: self # RA form project A
      type: git
      ref: $(Build.SourceBranch)
    - repository: mysub
      type: git
      name: B/RB
      ref: main # Branch to checkout

I can checkout both in the steps:

  steps:
  - checkout: self
    clean: true
    path: SomeFolder
    workspaceRepo: true # sets $(System.DefaultWorkingDirectory) to \D\a\1\SomeFolder
    
  - checkout: mysub
    path:$(System.DefaultWorkingDirectory)/mysub # Checkout mysub into \D\a\SomeFolder\mysub folder

This kind of defeats the purpose of submodules as it will always get the main branch of mysub, but it works, and proves that access to both projects/repos is granted.

When I change the steps section to use the submodule as a submodule following the commit hash of the RA:

  steps:
  - checkout: self
    clean: true
    submodules: true
    path: SomeFolder
    workspaceRepo: true # sets $(System.DefaultWorkingDirectory) to \D\a\1\SomeFolder

I get this error:

HEAD is now at d18c3a1 Trying to use submodules
git submodule sync
git --config-env=http.https://ORGANIZATION.visualstudio.com.extraheader=env_var_http.https://ORGANIZATION.visualstudio.com.extraheader submodule update --init --force --depth=1
Submodule 'mysub' (https://ORGANIZATION@dev.azure.com/ORGANIZATION/B/_git/RB) registered for path 'mysub'
Cloning into 'D:/a/1/SomeFolder/mysub'...
fatal: Cannot prompt because user interactivity has been disabled.
fatal: Cannot prompt because user interactivity has been disabled.
fatal: could not read Password for 'https://ORGANIZATION@dev.azure.com/ORGANIZATION/B/_git/RB': terminal prompts disabled
fatal: clone of 'https://ORGANIZATION@dev.azure.com/ORGANIZATION/B/_git/RB' into submodule path 'D:/a/1/SomeFolder/mysub' failed
Failed to clone 'mysub'. Retry scheduled

I have deselected the 'Limit job authorization scope to current project for release pipelines' at organization level as well as on project level:

enter image description here

EDIT: This is my .gitmodules file:

[submodule "mysub"]
    path = mysub
    url = https://ORGANIZATION@dev.azure.com/ORGANIZATION/B/_git/RB

I've also tried adding persistCredentials, so the step now looks like this:

  steps:
  - checkout: self
    clean: true
    persistCredentials: true
    submodules: true
    path: SomeFolder
    workspaceRepo: true # sets $(System.DefaultWorkingDirectory) to \D\a\1\SomeFolder

The error remains the same, except the indication of persistCredentials in first line

EDIT: I am new to YAML pipelines, but have several defined in the classic drag-and-drop style that clones with submodules without issues.

I should say that on organization level we still have not switched to the 'new' url https://dev.azure.com/ORGANIZATION, so we are still on https://ORGANIZATION.visualstudio.com, but in most cases the url's are interchangeable.

EDIT: The url in combination with the setting pointed out by Kevin Lu (Protect access to repositories in YAML pipelines) indeed was the key to succes. As we are still unable to switch to the 'new' url, changing the .gitmodules file back to the original together with the YAML option did the trick:

[submodule "mysub"]
    path = mysub
    url = https://ORGANIZATION.visualstudio.com/B/_git/RB
Solution

  • Submodule 'subfolder' (https://ORGANIZATION@dev.azure.com/ORGANIZATION/B/_git/RB) registered for path 'subfolder'

    From the error log, you have set the valid submodule git URL in the repo.

    Based on your confirmation, you have disabled the option: Limit job authorization scope to current project for non-release pipelines and set the required permissions to the submodule repo.

    Since you still encounter the error when checkout the submodule, you also need to disable the option: Protect access to repositories in YAML pipelines in Project A -> Project Settings -> Settings.

    For example:

    enter image description here

    When the option is enabled, it will generate a job access token that is scoped to repositories that are explicitly referenced in the YAML pipeline.

    This could be the reason why you can successfully reference Repo B resource (via repo resources) directly in the Yaml pipeline.

    For more detailed info, you can refer to this doc: Protect access to repositories in YAML pipelines

    Update:

    From your latest update of the question, your organization has disabled the new URL(dev.azure.com/xxx) and use old URL(xxx.visualstudio.com). Tested the same settings and I can reproduce the same issue.

    To solve this issue, you can enable the New Organization URL in Organization Settings.

    Or you can change the repo url in the .gitmodules file to old url.

    For example:

    [submodule "mysub"]
    path = mysub
    url = https://ORGANIZATION.visualstuido.com/B/_git/RB