I have configured my gitlab runner to work with the kubernetes pod. I see that my angular build job gets stuck at below and it keeps going on
Warning: Worker #10 exited unexpectedly (code: null | signal: SIGKILL).
Current task: {entryPoint: @angular/forms, formatProperty: es2015, processDts: Yes}
Current phase: compiling
I believe it may be because of a memory/cpu issue. But I am not able to debug as to what would be the issue.
Below are my values.yaml file I used for setting up runners. But below mentioned cpu/memory are not picked up while runners are running.
## GitLab Runner Image
##
## By default it's using gitlab/gitlab-runner:alpine-v{VERSION}
## where {VERSION} is taken from Chart.yaml from appVersion field
##
## ref: https://hub.docker.com/r/gitlab/gitlab-runner/tags/
##
image:
registry:<artifactory>/docker-images/gitlab
image: gitlab-runner
tag: alpine-v13.1.0
## Specify a imagePullPolicy
## 'Always' if imageTag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
imagePullPolicy: Always
## The GitLab Server URL (with protocol) that want to register the runner against
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register
##
gitlabUrl: <gitlab-url>
## The Registration Token for adding new Runners to the GitLab Server. This must
## be retrieved from your GitLab Instance.
## ref: https://docs.gitlab.com/ce/ci/runners/README.html
##
runnerRegistrationToken: "sdsdsdsdsfsfsfsd"
## The Runner Token for adding new Runners to the GitLab Server. This must
## be retrieved from your GitLab Instance. It is token of already registered runner.
## ref: (we don't yet have docs for that, but we want to use existing token)
##
# runnerToken: ""
#
## Unregister all runners before termination
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated and created again. This may cause your Gitlab instance to reference
## non-existant runners. Un-registering the runner before termination mitigates this issue.
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-unregister
##
unregisterRunners: true
## When stopping the runner, give it time to wait for its jobs to terminate.
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated with a graceful stop request. terminationGracePeriodSeconds
## instructs Kubernetes to wait long enough for the runner pod to terminate gracefully.
## ref: https://docs.gitlab.com/runner/commands/#signals
terminationGracePeriodSeconds: 9000
## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use
## Provide resource name for a Kubernetes Secret Object in the same namespace,
## this is used to populate the /home/gitlab-runner/.gitlab-runner/certs/ directory
## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates
##
# certsSecretName:
## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 10
## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 30
## Configure GitLab Runner's logging level. Available values are: debug, info, warn, error, fatal, panic
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
# logLevel:
## Configure GitLab Runner's logging format. Available values are: runner, text, json
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
# logFormat:
## For RBAC support:
rbac:
create: false
## Define specific rbac permissions.
# resources: ["pods", "pods/exec", "secrets"]
# verbs: ["get", "list", "watch", "create", "patch", "delete"]
## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs
## cluster-wide or only within namespace
clusterWideAccess: false
## Use the following Kubernetes Service Account name if RBAC is disabled in this Helm chart (see rbac.create)
##
serviceAccountName: xyz-prod-admin
## Specify annotations for Service Accounts, useful for annotations such as eks.amazonaws.com/role-arn
##
## ref: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
##
# serviceAccountAnnotations: {}
## Configure integrated Prometheus metrics exporter
## ref: https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server
metrics:
enabled: false
## Configuration for the Pods that that the runner launches for each new job
##
runners:
## Default container image to use for builds when none is specified
##
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
## Specify one or more imagePullSecrets
##
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# imagePullSecrets: []
## Specify the image pull policy: never, if-not-present, always. The cluster default will be used if not set.
##
imagePullPolicy: "always"
## Defines number of concurrent requests for new job from GitLab
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
##
# requestConcurrency: 1
## Specify whether the runner should be locked to a specific project: true, false. Defaults to true.
##
# locked: true
## Specify the tags associated with the runner. Comma-separated list of tags.
##
## ref: https://docs.gitlab.com/ce/ci/runners/#using-tags
##
# tags: ""
## Specify if jobs without tags should be run.
## If not specified, Runner will default to true if no tags were specified. In other case it will
## default to false.
##
## ref: https://docs.gitlab.com/ce/ci/runners/#allowing-runners-with-tags-to-pick-jobs-without-tags
##
runUntagged: true
## Specify whether the runner should only run protected branches.
## Defaults to False.
##
## ref: https://docs.gitlab.com/ee/ci/runners/#protected-runners
##
# protected: true
## The name of the secret containing runner-token and runner-registration-token
# secret: gitlab-runner
## Namespace to run Kubernetes jobs in (defaults to the same namespace of this release)
##
# namespace:
## The amount of time, in seconds, that needs to pass before the runner will
## timeout attempting to connect to the container it has just created.
## ref: https://docs.gitlab.com/runner/executors/kubernetes.html
pollTimeout: 180
## Set maximum build log size in kilobytes, by default set to 4096 (4MB)
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
outputLimit: 4096
## Distributed runners caching
## ref: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/configuration/autoscale.md#distributed-runners-caching
##
## If you want to use s3 based distributing caching:
## First of all you need to uncomment General settings and S3 settings sections.
##
## Create a secret 's3access' containing 'accesskey' & 'secretkey'
## ref: https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/
##
## $ kubectl create secret generic s3access \
## --from-literal=accesskey="YourAccessKey" \
## --from-literal=secretkey="YourSecretKey"
## ref: https://kubernetes.io/docs/concepts/configuration/secret/
##
## If you want to use gcs based distributing caching:
## First of all you need to uncomment General settings and GCS settings sections.
##
## Access using credentials file:
## Create a secret 'google-application-credentials' containing your application credentials file.
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscachegcs-section
## You could configure
## $ kubectl create secret generic google-application-credentials \
## --from-file=gcs-application-credentials-file=./path-to-your-google-application-credentials-file.json
## ref: https://kubernetes.io/docs/concepts/configuration/secret/
##
## Access using access-id and private-key:
## Create a secret 'gcsaccess' containing 'gcs-access-id' & 'gcs-private-key'.
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-cache-gcs-section
## You could configure
## $ kubectl create secret generic gcsaccess \
## --from-literal=gcs-access-id="YourAccessID" \
## --from-literal=gcs-private-key="YourPrivateKey"
## ref: https://kubernetes.io/docs/concepts/configuration/secret/
cache: {}
## General settings
# cacheType: s3
# cachePath: "gitlab_runner"
# cacheShared: true
## S3 settings
# s3ServerAddress: s3.amazonaws.com
# s3BucketName:
# s3BucketLocation:
# s3CacheInsecure: false
# secretName: s3access
## GCS settings
# gcsBucketName:
## Use this line for access using access-id and private-key
# secretName: gcsaccess
## Use this line for access using google-application-credentials file
# secretName: google-application-credentials
## Build Container specific configuration
##
builds:
cpuLimit: 3
cpuLimitOverwriteMaxAllowed: 3
memoryLimit: 2Gi
memoryLimitOverwriteMaxAllowed: 5Gi
cpuRequests: 2
cpuRequestsOverwriteMaxAllowed: 2
memoryRequests: 1Gi
memoryRequestsOverwriteMaxAllowed: 2Gi
## Service Container specific configuration
##
services:
cpuLimit: 3
memoryLimit: 4Gi
cpuRequests: 2
memoryRequests: 2Gi
## Helper Container specific configuration
##
helpers:
cpuLimit: 512m
memoryLimit: 512Mi
cpuRequests: 512m
memoryRequests: 512Mi
image: "<artifactory>/docker-images/gitlab/gitlab-runner-helper:x86_64-6214287e"
## Helper container security context configuration
## Refer to https://docs.gitlab.com/runner/executors/kubernetes.html#using-security-context
# pod_security_context:
# run_as_non_root: true
# run_as_user: 100
# run_as_group: 100
# fs_group: 65533
# supplemental_groups: [101, 102]
## Service Account to be used for runners
##
# serviceAccountName:
## If Gitlab is not reachable through $CI_SERVER_URL
##
# cloneUrl:
## Specify node labels for CI job pods assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
##
# nodeSelector: {}
## Specify node tolerations for CI job pods assignment
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
# nodeTolerations: {}
## Specify pod labels for CI job pods
##
# podLabels: {}
## Specify annotations for job pods, useful for annotations such as iam.amazonaws.com/role
# podAnnotations: {}
## Configure environment variables that will be injected to the pods that are created while
## the build is running. These variables are passed as parameters, i.e. `--env "NAME=VALUE"`,
## to `gitlab-runner register` command.
##
## Note that `envVars` (see below) are only present in the runner pod, not the pods that are
## created for each build.
##
## ref: https://docs.gitlab.com/runner/commands/#gitlab-runner-register
##
env:
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
## Configure securitycontext
## ref: http://kubernetes.io/docs/user-guide/security-context/
##
securityContext:
# fsGroup: 65533
runAsUser: 100
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
limits:
memory: 5Gi
cpu: 2
requests:
memory: 2Gi
cpu: 1
## Affinity for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
# Example: The gitlab runner manager should not run on spot instances so you can assign
# them to the regular worker nodes only.
# node-role.kubernetes.io/worker: "true"
## List of node taints to tolerate (requires Kubernetes >= 1.6)
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: []
# Example: Regular worker nodes may have a taint, thus you need to tolerate the taint
# when you assign the gitlab runner manager with nodeSelector or affinity to the nodes.
# - key: "node-role.kubernetes.io/worker"
# operator: "Exists"
## Configure environment variables that will be present when the registration command runs
## This provides further control over the registration process and the config.toml file
## ref: `gitlab-runner register --help`
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
##
# envVars:
# - name: RUNNER_EXECUTOR
# value: kubernetes
## list of hosts and IPs that will be injected into the pod's hosts file
hostAliases: []
# Example:
# - ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
# - ip: "10.1.2.3"
# hostnames:
# - "foo.remote"
# - "bar.remote"
## Annotations to be added to manager pod
##
podAnnotations: {}
# Example:
# iam.amazonaws.com/role: <my_role_arn>
## Labels to be added to manager pod
##
podLabels: {}
# Example:
# owner.team: <my_cool_team>
## HPA support for custom metrics:
## This section enables runners to autoscale based on defined custom metrics.
## In order to use this functionality, Need to enable a custom metrics API server by
## implementing "custom.metrics.k8s.io" using supported third party adapter
## Example: https://github.com/directxman12/k8s-prometheus-adapter
##
#hpa: {}
# minReplicas: 1
# maxReplicas: 10
# metrics:
# - type: Pods
# pods:
# metricName: gitlab_runner_jobs
# targetAverageValue: 400m
The resources
that you provided in your values file are only for the runner itself and not the jobs it produces when a pipeline runs. To increase the cpu/memory requests/limits you need to provide additional configuration. In this additional configuration, you can set the requests and limits in the same way you would with a config.toml file.
You're really just missing something like the following:
runners:
config: |
[[runners]]
executor = "kubernetes"
[runners.kubernetes]
cpu_request = 1
cpu_limit = 2
memory_request = 2Gi
memory_limit = 5Gi
There are also a lot more config options in the second link so it might be good to look thru and see if there's any additional options that would help out.