azureazure-devops

Azure DevOps Pipeline YAML: Repository Resource URL with Endpoint


I have a pipeline which should pull from another repository:


pool:
  vmImage: ubuntu-latest

resources:
  repositories:
    - repository: upstream
      type: git
      name: foo/bar
      ref: refs/heads/master
      endpoint: upstream-access

variables:
  repourl: $[ resources.repositories.upstream.url ]

steps:
- checkout: self
  persistCredentials: true
  clean: true

- script: |
    git branch -va
    git remote add upstream-repo $(repourl) || true
    git fetch upstream-repo
    # Check if upstream branch exists locally; create it if it doesn’t
    if git rev-parse --verify upstream >/dev/null 2>&1; then
      git checkout upstream
    else
      git checkout -b upstream upstream-repo/master
      git push origin upstream  # Initial push to create the branch on origin
    fi
    git pull upstream-repo master --rebase
    git push origin upstream
  displayName: 'Sync upstream branch with $(repourl)'

But this would generate a URL like this:

Job preparation parameters
Variables:
  repourl:
    Parsing expression: <resources.repositories.upstream.url>
    Evaluating: resources['repositories']['upstream']['url']
    Result: 'https://dev.azure.com/orgname/840afda9-cc92-42ac-aeb4-fee9f56de5a0/_apis/git/repositories/972c972e-5c39-4420-b779-ec482edb0aa9/'
ContinueOnError: False
TimeoutInMinutes: 60
CancelTimeoutInMinutes: 5
Expand:
  MaxConcurrency: 0
...
fatal: repository 'https://dev.azure.com/orgname/840afda9-cc92-42ac-aeb4-fee9f56de5a0/_apis/git/repositories/972c972e-5c39-4420-b779-ec482edb0aa9/' not found
fatal: 'upstream-repo/master' is not a commit and a branch 'upstream' cannot be created from it
error: src refspec upstream does not match any
...

However, if I just remove the endpoint from resources.repositories.upstream, the URL would be correctly formed:

Job preparation parameters
Variables:
  repourl:
    Parsing expression: <resources.repositories.upstream.url>
    Evaluating: resources['repositories']['upstream']['url']
    Result: 'https://orgname@dev.azure.com/orgname/prjname/_git/reponame'
ContinueOnError: False
TimeoutInMinutes: 60
CancelTimeoutInMinutes: 5
Expand:
  MaxConcurrency: 0
fatal: could not read Password for 'https://orgname@dev.azure.com': terminal prompts disabled
fatal: 'upstream-repo/master' is not a commit and a branch 'upstream' cannot be created from it
error: src refspec upstream does not match any

Questions:

  1. Is this a bug? If yes, what would be the workaround?
  2. What's the best way to access another repository from a pipeline? In my case both repos are in the same organization, not in the same project though.

Solution

  • Update

    Please note that the error fatal: could not read Password for 'https://orgname@dev.azure.com in your pipeline was NOT a permission error.

    Your pipeline only checked out the self repo without checkout: upstream. Since persistCredentials was set as true, the checkout:self step was running and added the $(System.AccessToken) in the .git/config file for your downstream git commands to use. But it was only stored there for authenticating access to your self repo.

    git config

    In the script, it simply added remote url to your upstream repo, but didn't tell the command to use a password.

    With those being said, you may add git config http.$(repourl).extraheader "AUTHORIZATION: bearer $(System.AccessToken)" before fetch. Here is a sample YAML pipeline.

    pool: # Default # Run the agent job on a self-hosted agent to check git config in the pipeline working directory.
      vmImage: ubuntu-latest
    
    resources:
      repositories:
      - repository: upstream
        type: git
        name: foo/bar
        ref: refs/heads/master
        # endpoint: upstream-access
    
    variables:
      repourl: $[ resources.repositories.upstream.url ]
    
    steps:
    - checkout: self
      persistCredentials: true
    
    - script: |
        echo "repourl - $(repourl)"
        
        git branch -va
        git remote add upstream-repo $(repourl) || true
        git config http.$(repourl).extraheader "AUTHORIZATION: bearer $(System.AccessToken)"
        git fetch upstream-repo
        
        sleep 120 # Pause the agent job to check the git config file on the self-hosted agent
    

    upstream

    Adding steps for your reference to check the required pipeline service account permissions.

    1. Navigate to your Organization Settings and Project Settings -> Pipelines -> Settings -> Disable Limit job authorization scope to current project for non-release pipelines; this option will have the build pipelines in this project to run as the pipeline service account Project Collection Build Service (<YourOrgName>) in order to access the repos outside this project.

      settings

    2. Check the Project Settings of foo -> Repos -> Repositories -> In the Security tab of repo bar -> Set the permission to Contribute as Allow for the pipeline service account Project Collection Build Service (<YourOrgName>); set other permissions for your pipeline service account based on the requirements of your git commands.

      security


    The endpoint that your upstream repo resource is referring to is likely the name of an Azure Repos service connection, which is typically used to connect to another Azure DevOps organization using basic or token-based authentication.

    When creating an Azure Repos service connection, you provide the organization URL along with token-based authentication.

    svc cnn

    When running a pipeline that defines a repository resource with an endpoint, it authenticates using the token and then searches in the connected organization (via url) to find the repository (bar) from the project (foo).

    variable

    In your scenario, since the pipeline is in the same Azure DevOps organization as the repository but in a different project, you don't need to authenticate using an endpoint. The pipeline's service account should already have the necessary permissions to access the repository.

    For more information, refer to the following documents:

    Check out multiple repositories in your pipeline

    Repository resource definition

    resources.repositories.repository YAML schema