[SOLVED] Gitlab job / docker execution hangs on node command forever

Gitlab job / docker execution hangs on node command forever

We are upgrading our Anguar project from v.17 to v.18.1.0. Everything works nicely, but we get a problem during our CI/CD pipelines.

I have the following commands to run as part of a gitlab pipeline:

image: node:21.4.0
...
    - npm run lint
    - npm run prettier

Since we upgraded to Angular 18.1, this script get stuck and never finishes (no exit code, it just hangs execution indefinetly).

In our gitlab logs:

Running with gitlab-runner 16.8.0 (c72a09b6)
  on docker-runner-63-48-#1 6Bbjxbz5, system ID: s_0f499b5c2941
Preparing the "docker" executor
00:03
Using Docker executor with image node:21.4.0 ...
Authenticating with credentials from /root/.docker/config.json
Pulling docker image node:21.4.0 ...
Using docker image sha256:5aeb975d4a5c53f930d848c6d40af3b11232bf7679acae74201c40d5136dc97c for node:21.4.0 with digest node@sha256:814a6dc5bfd4ecc5ef24652f6b4f2790e9f3552b52ee38a7b51fc4d4c0d6d7fd ...
Preparing environment
00:00
Running on runner-6bbjxbz5-project-705-concurrent-0 via oraclegitlab...
Getting source from Git repository
00:05
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/.../customer-webportal-frontend/.git/
Checking out b1c0df15 as detached HEAD (ref is feature/819-angular-update)...
Removing node_modules/
Removing package-lock.json
Skipping Git submodules setup
Downloading artifacts
00:25
Downloading artifacts for install (818083)...
Downloading artifacts from coordinator... ok        host=gitlab.....hu id=818083 responseStatus=200 OK token=glcbt-64
Executing "step_script" stage of the job script
54:34
Using docker image sha256:5aeb975d4a5c53f930d848c6d40af3b11232bf7679acae74201c40d5136dc97c for node:21.4.0 with digest node@sha256:814a6dc5bfd4ecc5ef24652f6b4f2790e9f3552b52ee38a7b51fc4d4c0d6d7fd ...
$ npm run lint
> customer-webportal-fe@1.0.2 lint
> ng lint --fix
Node.js version v21.4.0 detected.
Odd numbered Node.js versions will not enter LTS status and should not be used for production. For more information, please see https://nodejs.org/en/about/previous-releases/.
Linting "customer-webportal-fe"...
All files pass linting.
(node:32) ExperimentalWarning: WASI is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
Terminated
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: script timeout context: context deadline exceeded

I tried to execute the same job with registering my PC as a runner (same settings, docker with node v.21.4.0). I get the same output: after the said ExperimentalWarning, it just hangs forever.

forest@forestg:~$ docker logs -f --tail 1000 1b4fcd4e2c32
$ npm run lint

> customer-webportal-fe@1.0.2 lint
> ng lint --fix

Node.js version v21.4.0 detected.
Odd numbered Node.js versions will not enter LTS status and should not be used for production. For more information, please see https://nodejs.org/en/about/previous-releases/.

Linting "customer-webportal-fe"...

All files pass linting.

(node:32) ExperimentalWarning: WASI is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

If I run these commands manually on the project, directly on my same machine where I have the docker running, but without docker or gitlab-runner, everything is fine. (it executes both commands in ~5 seconds)

forest@forestg:~/projects/.../customer-webportal-frontend$ npm run lint && npm run prettier

> customer-webportal-fe@1.0.2 lint
> ng lint --fix

Node.js version v21.4.0 detected.
Odd numbered Node.js versions will not enter LTS status and should not be used for production. For more information, please see https://nodejs.org/en/about/previous-releases/.

Linting "customer-webportal-fe"...

All files pass linting.


> customer-webportal-fe@1.0.2 prettier
> pretty-quick

🔍  Finding changed files since git revision null.
🎯  Found 1 changed file.
✅  Everything is awesome!

The same happens if I upgrade the node version to 22.0.0. What can be the problem here?

Solution

None of the above answers helped. We disabled the caching steps for the given job, and the problem went away. We have 0 clue how can that relate to the problem - maybe shared cache between multiple gitlab runner machines can cause this phenomenom.