I am using GitLab ci to run SonarCloud code analysis on the code.
here is my gitlab-ci.yaml
stages:
- test
before_script:
- mkdir -p ~/.ssh &&
cp $gitlab_private_key ~/.ssh/id_ed25519 &&
chmod 600 ~/.ssh/id_ed25519 &&
touch ~/.ssh/known_hosts &&
ssh-keyscan gitlab.com >> ~/.ssh/``known_hosts
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar" # Defines the location of the analysis task cache
GIT_DEPTH: "0" # Tells git to fetch all the branches of the project, required by the analysis task
GITLAB_PROJECT_ID: ${CI_PROJECT_ID} # needed to be exported to the project's environments
FLASK_APP: manage.py
sonarcloud-check:
image:
name: sonarsource/sonar-scanner-cli:latest
entrypoint: [""]
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
script:
- sonar-scanner
only:
- merge_requests
- master
test-merge-request-changes:
stage: test
only:
- merge_requests
image:
name: docker:19.03.13-git
services:
- name: docker:19.03.0-dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh"]
variables:
DOCKER_HOST: tcp://localhost:2375
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
ENV: test
CI_DEBUG_TRACE: "true"
before_script:
- echo $CI_BUILD_TOKEN | docker login -u gitlab-ci-token --password-stdin ${CI_REGISTRY}
script:
- echo "Running Tests..."
- cp ${group_shared_vars} ${CI_PROJECT_DIR}/.env
- docker build . -f Dockerfile-testing -t test_merge_req --build-arg GITLAB_PROJECT_ID=${GITLAB_PROJECT_ID}
- docker run --cidfile="my-package.txt" test_merge_req:latest
after_script:
- touch text2.txt
- docker cp $(cat my-package.txt):/app/tests/coverage/coverage.xml coverage.xml
- docker cp $(cat my-package.txt):/app/tests/coverage/junit.xml junit.xml
timeout: 2h
artifacts:
when: always
reports:
cobertura:
- coverage.xml
junit:
- junit.xml
coverage: '/TOTAL.*\s+(\d+%)$/'
And here is my sonar-project.properties
sonar.projectKey=my_app-key
sonar.organization=my_org
sonar.sources=lib
sonar.tests=tests
sonar.exclusions=tests
sonar.language=python
sonar.python.version=3.8
I want to get the report that is generated in the container analyzed by sonarcloud on each merge request.
Also, when a code is pushed to the master branch, I want to get the coverage percent on sonarcloud of the project to be updated but it just shows 0%.
Is there any way that after the merge requests are run, we get the sonarcloud analysis on the report of the docker container?
And also getting the master branch coverage updated without having to commit the coverage.xml to the repo?
After digging into how Gitlab CI stages and jobs work and also the insight that this thread brought, I have tweaked the above GitLab ci so that it :
path that was replaced by pytest-cov in the <source>...</source> with a sed command. (This step is due to the path that was placed into the source element is relevant to the docker container and then after downloading the coverage report not working inside the GitLab ci container)sonar-scanner on the coverage report.GitLab ci:
stages:
- build_test
- analyze_test
before_script:
- mkdir -p ~/.ssh &&
cp $gitlab_private_key ~/.ssh/id_ed25519 &&
chmod 600 ~/.ssh/id_ed25519 &&
touch ~/.ssh/known_hosts &&
ssh-keyscan gitlab.com >> ~/.ssh/``known_hosts
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar" # Defines the location of the analysis task cache
GIT_DEPTH: "0" # Tells git to fetch all the branches of the project, required by the analysis task
GITLAB_PROJECT_ID: ${CI_PROJECT_ID} # needed to be exported to the projects environments
include:
- template: Code-Quality.gitlab-ci.yml
test-merge-request:
stage: build_test
only:
- merge_requests
- master
image:
name: docker:19.03.13-git
cache:
key: "${CI_JOB_NAME}"
paths:
- build
services:
- name: docker:19.03.0-dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh"]
variables:
DOCKER_HOST: tcp://localhost:2375
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
ENV: test
CI_DEBUG_TRACE: "true"
before_script:
- echo $CI_BUILD_TOKEN | docker login -u gitlab-ci-token --password-stdin ${CI_REGISTRY}
script:
- echo "Running Tests..."
- cp ${group_shared_vars} ${CI_PROJECT_DIR}/.env
- docker build . -f Dockerfile-testing -t test_merge_req --build-arg GITLAB_PROJECT_ID=${GITLAB_PROJECT_ID}
- docker run --cidfile="my-package.txt" test_merge_req:latest
after_script:
- docker cp $(cat my-package.txt):/app/tests/coverage/coverage.xml build/coverage.xml
- docker cp $(cat my-package.txt):/app/tests/coverage/junit.xml build/junit.xml
timeout: 2h
artifacts:
when: always
paths:
- build
reports:
cobertura:
- build/coverage.xml
junit:
- build/junit.xml
expire_in: 30 min
coverage: '/TOTAL.*\s+(\d+%)$/'
sonarcloud-check:
stage: analyze_test
image:
name: sonarsource/sonar-scanner-cli:latest
entrypoint: [""]
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
script:
- echo "Logging coverage..."
- sed -i "s|<source>\/app\/my_app<\/source>|<source>$CI_PROJECT_DIR\/my_app<\/source>|g" ./build/coverage.xml
- sonar-scanner
only:
- merge_requests
- master
dependencies:
- test-merge-request-changes
The takeaway from my issue was that Gitlab doesn't share the artifacts between the jobs that are placed in one GitLab ci stage, but rather shares them between different stages.
So, I just created two separate stages, one for building the tests (build_test) and one for analyzing the tests (analyze_test). Then, made the sonar-cloud job dependent on the test-merge-request-changes job. This way, we make sure that first the tests runs in the test-merge-request-changes and then the uploaded artifacts are utilized inside the sonar-cloud stage.