Unable to get ENV variables in GoCD Kubernetes using YAML config

GoCD Version: 19.12.0

I'm trying to get environment variables defined in the Kubernetes deployment (system) in my GoCD YAML config in order to pass the GitHub authentication when pulling the resource. I've confirmed that I'm able to call the repository using a personal access token. (via https://[TOKEN]@github.com/[COMPANY]/[REPO].git)

This, of course, also works if I do the same for the actual YAML git field.

The GoCD secrets in K8s:

apiVersion: v1
data:
  GITHUB_ACCESS_KEY: base64EncodedKey
kind: Secret
type: Opaque

The GoCD deployment gets the secrets:

...
spec:
  containers:
  - env:
   - name: GOCD_PLUGIN_INSTALL_kubernetes-elastic-agents
     value: https://github.com/gocd/kubernetes-elastic-agents/releases/download/v3.4.0-196/kubernetes-elastic-agent-3.4.0-196.jar
   - name: GOCD_PLUGIN_INSTALL_docker-registry-artifact-plugin
     value: https://github.com/gocd/docker-registry-artifact-plugin/releases/download/v1.1.0-104/docker-registry-artifact-plugin-1.1.0-104.jar
   - name: GITHUB_ACCESS_KEY
     valueFrom:
       secretKeyRef:
         key: GITHUB_ACCESS_KEY
         name: gocd-server
...

I've exec'd into the pod and echoed the variable, which returns the decoded value.

The YAML:

format_version: 9
pipelines:
  db-docker-build:
    group: someGroup
    label_template: ${COUNT}-${git[:8]}
    lock_behavior: unlockWhenFinished
    display_order: 1
    materials:
      git:
        git: 'https://$GITHUB_ACCESS_KEY@github.com/[COMPANY]/[REPO].git'
        shallow_clone: true
        auto_update: true
        branch: master
...

I'd half expect that to work, but it doesn't, it actually just gets $GITHUB_ACCESS_KEY as the value. The jobs defined in the pipeline stages are run using an elastic agent pod which also has the required secrets defined. I've tried a few

Setting env variables -

environment_variables: GIT_KEY: ${GITHUB_ACCESS_KEY}

and then using that variable

git: 'https://$GIT_KEY@github.com/[COMPANY]/[REPO].git'

Setting env variables and no quotes -

environment_variables: GIT_KEY: ${GITHUB_ACCESS_KEY}

and then using that variable

git: https://${GIT_KEY}@github.com/[COMPANY]/[REPO].git

No quotes - git: https://$GITHUB_ACCESS_KEY@github.com/[COMPANY]/[REPO].git

No quotes with brackets - git: https://${GITHUB_ACCESS_KEY}@github.com/[COMPANY]/[REPO].git

I've seen from some YAML documentation that it is recommended to use encrypted_password for the GitHub password, but this seems unnecessary since the GUI hides the token, and that its running in Kubernetes with secrets.

Solution

The team and I researched this a little further and found a workaround. Most issues and articles explain what is written in the docs, that you really need access to /bin/bash -c in order to get the variables.

The YAML plugin creator also uses secure, encrypted variables to store sensitive data which is fine, but for our team it means that a lot of Kubernetes features are not utilised.

The workaround:

Use the GUI to create a pipeline in GoCD, enter the GitHub link, add a username and the personal access token for the user as the password, test the connection is OK. Once created, go to Admin -> Pipelines and click the Download pipeline configuration and select YAML.

The generated YAML has the token encrypted as with the GoCD servers private key.