gitlabpipelinecicd

Gitlab multi project pipeline - This job could not start because it could not retrieve the needed artifacts


So I have another project which contains artifacts I want to reuse on my project/job/pipeline. So I try to trigger the "another-project"'s pipeline to build the artifacts which I want to reuse in my job of my pipeline. I tried different setups:

CHILD pipeline (another-group/another-project/.gitlab-ci.yml)

stages:
  - build

build:
  image: node:latest
  stage: build
  script:
  # Resolves to typescript npx tsc --build
  - yarn run build
  artifacts:
    name: "${CI_PROJECT_NAME}-${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHORT_SHA}"
    when: always
    paths:
    - $CI_PROJECT_DIR/dist/
expire_in: 90 day

1st version

I tried to use trigger to trigger the child pipeline and then depend on it with needs: - job. The pipeline of the child is running. I can download the artifacts of build, but somehow it is not passed upwards. There is no errror, but also no artifacts.

PARENT pipeline (my-group/my-project/.gitlab-ci.yml)

trigger-child:
  stage: .pre
  trigger:
    project: another-group/another-project
    branch: main
    strategy: depend


parent-job:
  stage: afterwards
  needs:
  # No job error, but artifacts aren't there aswell...
  - job: trigger-child
    artifacts: true
  # Job error (see title)
  - project: another-group/another-project
    job: build
    ref: main
    artifacts: true

2nd version

I tried the needs: - pipeline. I expected this version to not trigger the child but get the newest artifact. The job fails with

This job could not start because it could not retrieve the needed artifacts. Learn more about dependencies and common causes of this error.

PARENT pipeline (my-group/my-project/.gitlab-ci.yml)

parent-job:
  stage: afterwards
  needs:
  - project: another-group/another-project
    job: build
    ref: main
    artifacts: true

How can I not only trigger the "child" pipeline but also reuse the artifacts when I run my job afterwards? I tried both needs approaches but neither of them works. Standalone or Combined.

of the Gitlab docs:

To download artifacts from a different pipeline in the current project, set project to be the same as the current project, but use a different ref than the current pipeline. Concurrent pipelines running on the same ref could override the artifacts.

The user running the pipeline must have at least the Reporter role for the group or project, or the group/project must have public visibility.

You can’t use needs:project in the same job as trigger.

When using needs:project to download artifacts from another pipeline, the job does not wait for the needed job to complete. Using needs to wait for jobs to complete is limited to jobs in the same pipeline. Make sure that the needed job in the other pipeline completes before the job that needs it tries to download the artifacts.

You can’t download artifacts from jobs that run in parallel.

https://docs.gitlab.com/ci/yaml/#needsproject

I have the "owner" role in both projects: role on gitlab

EDIT: I cared to elaborate!

EDIT2: I found a similar topic on the official Gitlab forum: https://forum.gitlab.com/t/using-artifacts-of-another-project/90260


Solution

  • I solved it with a workaround. I manually download the artifacts after I have triggered the pipeline.

    trigger-child:
      stage: .pre
      trigger:
        project: another-group/another-project
        branch: main
        strategy: depend
    
    # You need to allow the CI_JOB_TOKEN to be used for the project you intend to download artifacts of
    # You can find this setting on Setting/"CI/CD"/Job Token Permissions
    gitlab:job:download-artifacts:
      stage: .pre
      needs:
      - job: trigger-child
      variables:
        PROJECT_ID: 12345 # the project id
        BRANCH: main
        JOB: build # the name of the job
        DOWNLOAD_URL: ${CI_API_V4_URL}/projects/${PROJECT_ID}/jobs/artifacts/${BRANCH}/download?job=${JOB}
        PATH_TO_EXTRACT: extracted-artifacts/
      script:
      - apt-get update && apt-get install -y curl unzip
      - echo "Trigger finished. Now downloading artifacts..."
      - |
        curl --header "JOB-TOKEN: $CI_JOB_TOKEN" "${DOWNLOAD_URL}" --output artifacts.zip
      - unzip artifacts.zip -d ${PATH_TO_EXTRACT}
      artifacts:
        paths:
        - ${PATH_TO_EXTRACT}
        expire_in: 1 week
    
    parent-job:
      stage: afterwards
      needs:
      - job: gitlab:job:download-artifacts
        artifacts: true
      variables: 
         DOWNLOADED_ARTIFACTS: $CI_PROJECT_DIR/extracted-artifacts/build.zip