cmakegitlabgitlab-ci

CMake fails parsing only when called in GitLab CI job with "Instead found unterminated bracket with text ..."

I have a repository that builds and supplies artifacts to other projects. In order to get the latest artifacts the GitLab Job Artifacts API offers two options based on the source of the request - inside a CI job or elsewhere. The URL I am using is

https://gitlab.example.com/api/v4/projects/${proj_id}/jobs/artifacts/${branch_tag}/download?job=${job_name}

I have a separate download.cmake files inside a cmake directory of my project (all of which is configured using CMake), where FetchContent_Declare() and FetchContent_MakeAvailable() are being executed to ensure that the project is setup for both development (locally) and deployment (on-premise premium GitLab instance).

Among other function inside download.cmake the following is what I use for downloading any artifact from my GitLab:

function(download_file_gitlab_latest name token proj_id branch_tag job_name)
    set(_URL "https://gitlab.example.com/api/v4/projects/${proj_id}/jobs/artifacts/${branch_tag}/download?job=${job_name}")
    set(_HEADER "PRIVATE-TOKEN: ${token}")
    message("GET Request: ${_URL}")
    message("Destination prefix: ${name}")

    FetchContent_Declare(
        ${name}
        #QUIET
        URL "${_URL}"
        HTTP_HEADER ${_HEADER}
        DOWNLOAD_NO_EXTRACT 0
        DOWNLOAD_EXTRACT_TIMESTAMP 1
        DOWNLOAD_NO_PROGRESS 0
    )

    if(NOT ${name}_POPULATED)
        FetchContent_MakeAvailable(${name})
    endif()
endfunction(download_file_gitlab_latest)

The header of the request contains an single parameter PRIVATE-TOKEN that contains a group/project/personal access token required for authenticating any request against the GitLab API (more on that parameter later on).

The token is supplied by a token.txt file, which has a single line with the respective token and is loaded in the CMakeLists.txt that calls the download function from above líke this:

file(READ "${CMAKE_SOURCE_DIR}/token.txt" TOKEN) #<---- the token is loaded from the token.txt and stored inside the TOKEN variable
...
download_file_gitlab_latest(
    "${DOWNLOAD_DIR_PREFIX}"
    ${TOKEN} #<---------------------------------------- supplied by token.txt
    68522
    "${config}"
    "build-all-${config}"
)

where config's value is a single value from CMAKE_CONFIGURATION_TYPES (debug, release etc.).

Both locally and in my CI job I configure the project (which also triggers the fetching of the artifacts) using

cmake -Bbuild -G "Visual Studio 16 2019" -S.

The only difference is the passing of CMAKE_CONFIGURATION_TYPES in the CI job to ensure that only the artifacts for the respective build type are downloaded.

Configuring the project locally shows no signs of a problem and the artifacts are downloaded, extracted, copied to a specific directory, included in the project and the build step is executed, producing a working EXE. However, remotely I am getting the following error:

CMake Error at D:/Software/CMake/share/cmake-3.30/Modules/FetchContent.cmake:1416:EVAL:1:
  Parse error.  Function missing ending ")".  Instead found unterminated
  bracket with text "JOB-TOKEN: ��g".
Call Stack (most recent call first):
  D:/Software/CMake/share/cmake-3.30/Modules/FetchContent.cmake:1416 (cmake_language)
  cmake/download.cmake:32 (FetchContent_Declare)
  CMakeLists.txt:41 (download_file_gitlab_latest)
CMake Error at D:/Software/CMake/share/cmake-3.30/Modules/FetchContent.cmake:1416 (cmake_language):
  cmake_language unknown error.
Call Stack (most recent call first):
  cmake/download.cmake:32 (FetchContent_Declare)
  CMakeLists.txt:41 (download_file_gitlab_latest)
-- Configuring incomplete, errors occurred!

Respectively the build steps never takes place.

There are two crucial steps that I have in my CI job that ensure the proper setup for the download and can be seen in the CI job below:

build-imgui-dx11-demo:
  stage: build
  rules:
    - if: $CI_COMMIT_REF_NAME == "main"
      allow_failure: true
  artifacts:
    untracked: true
    paths:
      - build
      - cmake                       # Allows me to verify that PRIVATE-TOKEN has been replaced with JOB-TOKEN
      - token.txt                   # Allows me to verify that the CI_JOB_TOKEN is stored in the token.txt and it's not just an empty file
    when: always                    # Allows me to see the artifacts even if the CI job fails
  before_script:
    - echo $CI_JOB_TOKEN > token.txt
    - |
      (Get-Content ./cmake/download.cmake).Replace('PRIVATE-TOKEN', 'JOB-TOKEN') | Set-Content ./cmake/download.cmake
  script:
    - cmake -Bbuild -G "Visual Studio 16 2019" -S. -DCMAKE_CONFIGURATION_TYPES="release"
    - cmake --build build --target ALL_BUILD --config release

For local download I use PRIVATE-TOKEN as also described in the download.cmake function. Since it's really bad practice to upload passwords and personal tokens to a repo, the token.txt is listed in .gitignore and therefore needs to be generated in the CI job

echo $CI_JOB_TOKEN > token.txt

Further, since the project is configured in a CI job, I also replace PRIVATE-TOKEN with JOB-TOKEN in the header for FetchContent_Declare(), which I have verified by checking the changed download.cmake that indeed has

set(_HEADER "JOB-TOKEN: ${token}")

Since I have a premium subscription I do know that I can use needs: project to interconnect multiple repositories and their artifacts. However, I would like to do it this way since it does not depend on premium features and (with minimal changes) mirrors the local setup.

UPDATE: First I added a message() inside my download function

message("Token: ${token}") # token is the parameter of the function and not the TOKEN variable where the contents of token.txt are stored!

but got empty message. Even Token: substring was gone.

I then added a similar message

message("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
message("------------------------Found token ${TOKEN}---------------------")
message("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

right after the file(READ "${CMAKE_SOURCE_DIR}/token.txt" TOKEN). A screen shot of the GitLab CI log can be seen below:

enter image description here

It appears that contents of token.txt, while correct, are not properly read. I downloaded a couple of versions of that file from different CI job failed runs and it always consists of a different token string and an empty second line. I do believe that the second terminating empty line is what messes up the processing of the token. I need to see how to write the CI_JOB_TOKEN to a TXT file without automatically adding a new line or use some option provided by file(READ ...) in CMake.

Solution

  • The problem here was that echoing a text to a file automatically adds a new line to it (similar to Out-File powershell command).

    I changed my CI job accordingly:

    - |
  $CI_JOB_TOKEN | Out-File token.txt -NoNewline
  (Get-Content ./cmake/download.cmake).Replace('PRIVATE-TOKEN', 'JOB-TOKEN') | Set-Content ./cmake/download.cmake

    Notice the -NoNewline parameter, which ensures that the file contains a single line (the token).

    While this fixes the issue, I made another change to make my CMake project configuration less prone to such an error (I am actually already checking if the file exists and is empty) by replacing file(READ ...) with

    file(
    STRINGS
    "${CMAKE_SOURCE_DIR}/token.txt"
    TOKEN
    LIMIT_COUNT 1
)

    According to the file() the STRINGS mode reads ASCII strings, which in this specific case - GitLab tokens - is not an issue. In addition the LIMIT_COUNT restricts how many strings are read (documentation doesn't state it but I assume that string == line). This change ensures that token.txt will not break the configuration if echoed or otherwise created locally/inside CI job.

    The failing of the download is also due to the project not being whitelisted in the project, where the request for download is targeted at. This can be accomplished by adding it to the Limit access to list.