gitlabgitlab-ci

Clone sub-modules (present on other Git server) on Gitlab CI


I have a Gitlab project, let's say parentRepo which has a reference to the sub-modules present on Github.

[submodule "submodules/child1"]
    path = submodules/child1
    url = https://github.com/my-org/child1.git
[submodule "submodules/child2"]
    path = submodules/child2
    url = https://github.com/my-org/child2.git

When, I am running Gitlab CI for my parentRepo, it is unable to clone the submodules from Github, since the repositories on Github are not publicly accessible and needs some auth creds to clone.

I am getting an error like this in my Gitlab CI,

fatal: could not read Username for 'https://github.com': No such device or address

I have tried cloning using below snippets in the before_script block of my build job,

- echo "https://${GITHUB_USER_NAME}:${GITHUB_TOKEN}@github.com"
- git config --global credential.helper store
- echo "https://${GITHUB_USER_NAME}:${GITHUB_TOKEN}@github.com" > ~/.git-credentials

Also, tried with this approach,

git config --global url."https://${GITHUB_USER_NAME}:${GITHUB_TOKEN}@github.com".insteadOf https://github.com

but nothing has worked so far.

I have also tried using pre_clone_script and pre_get_sources_script hooks, but those are not supported, since I am using Gitlab 14.9.5-ee.

I am hopeful, there should be a way to get this working, since Gitlab supports sub-modules with both absolute and relative URLs, https://docs.gitlab.com/ee/ci/runners/git_submodules.html

but then also found some open issues here:

Any help on this is highly appreciated!


Solution

  • I finally had to resort to native git commands to get this working, since this has been an open issue at Gitlab here

    I eventually got it working by following steps:

     - Disabled Gitlab CI default clone, by using GIT_SUBMODULE_STRATEGY: none
     - Set up the credentials for Github
     - Added native git commands to clone sub-modules
    

    Here is some code snippet from gitlab-ci.yml

    variables:
      GIT_SUBMODULE_STRATEGY: none
    
    .clone-script-before: &clone-script-before
       - git config --global credential.helper store
       - echo "https://${GITHUB_USER_NAME}:${GITHUB_TOKEN}@github.com" > ~/.git-credentials
       - git submodule sync --recursive
       - git submodule update --init --recursive
    
    Build:
      variables:
        GIT_SUBMODULE_STRATEGY: none
      stage: build
      script:
        - *clone-script-before
        - MORE BUILD STEPS