gitlab-citestcafe

How to set up GitLab to run either full regression tests or subset based on meta tag and GitLab variable

I have been struggling to filter my GitLab so that it runs either a full regression or a subset of smoke tests depending on the TestCafe meta tag.

Info on the meta tag can be seen here.

So I have tagged some fixtures like this:

fixture`Login Test`.page`${config.ui.copUrl}`
  .meta('virtual-test-pack', 'true')

fixture`Logout Test`.page`${config.ui.copUrl}`
  .meta('virtual-test-pack', 'true')

I then have a package.json file that contains the jobs:

    "ci:cop:login": "start-server-and-test start-server http://localhost:xxxx/mgmt/health 'SUITE=Cop/Login TESTS=tests/cop/login.test.ts node test-runner.js'",
    "ci:cop:logout": "start-server-and-test start-server http://localhost:xxxx/mgmt/health 'SUITE=Cop/Login TESTS=tests/cop/logout.test.ts node test-runner.js'",

So what I want to do is I have a variable known as SMOKE_TESTS. If this variable is set to true, then run all tests for each job that contains the above meta tag. If it's false, then run all the tests for each job.

But I also want the pipeline to be clever so that if it's only running a subset of tests, it doesn't show all test jobs, only the jobs that contain meta tags in their tests. Also a test job can run tests from multiple files currently so I thought of mentioning that as the two jobs I provided above only contain single test file each.

How can this be achieved? Below is my GitLab file:

    stages:
      - device-healthcheck
      - cop-tests
      - pos-tests
    
    
    variables:
      SMOKE_TESTS:
        value: "false"
        options:
          - "false"
          - "true"

integration-test:
  allow_failure: true
  image: xxx
  before_script:
    - https_proxy=http://webproxy:xxx
    - yarn config set strict-ssl false
    - yarn config set caFile /usr/local/share/xxx/xxx.io-root-ca-xxx.crt 
    - yarn install
    - mkdir -p ./reports
  script:
    - yarn config set strict-ssl false
    - yarn config set caFile /usr/local/share/xxx/xxx.io-root-ca-xxx.crt 
    - yarn run ${TEST_SUITE:='test'}
    - ls -R ./reports || echo "No reports found"
    - cp -r ./reports /cwd/
  artifacts:
    reports:
      junit: reports/xunit.xml
    paths:
      - artifacts/videos/*.mp4
    when: always

cop-login:
  stage: cop-tests
  variables:
    TEST_SUITE: "ci:cop:login"
  needs: ["cop-healthcheck"]
  extends:
    - .integration-test

cop-logout:
  stage: cop-tests
  variables:
    TEST_SUITE: "ci:cop:logout"
  needs: ["cop-healthcheck"]
  extends:
    - .integration-test
Solution

  • You can add rules to your pipeline. What we have here is the ability to configure the executed jobs for a merge request using labels. In a job, we then have:

      rules:
    # make this job a mandatory part of a merge request
    - if: $CI_MERGE_REQUEST_LABELS =~ /(^|,)ci-include-testsuite-foo(,|$)/
    # make this job optional but don't block merge requests
    - when: manual
      allow_failure: true

    Notes: