[SOLVED] "Text file busy" during a docker build

I am seeing what I would consider a weird docker build issue. It seems to be pretty specific to my PC as I have seen this build properly on other machines. It looks like a docker build is failing with npm ERR! sh: 1: node-pre-gyp: Text file busy during an npm install command. I would have expected the build to complete without issues. If any issues arise, I expected them to be consistent across different machines - after all it's a docker build. It's also worth noting that this used to work properly on my PC and it broke only recently. I am not sure what changed. I don't think I did any dramatic docker-related changes in the recent days.

I am looking for any advise on what to try to fix this issue. I will probably resort to reinstalling docker in a few days. Either way, I consider this to be an interesting issue and perhaps discussing it here will help someone else.

Below is a minimal setup which fails on my PC but works on others.

package.json:

{
  "name": "test",
  "version": "0.1.0",
  "author": "test",
  "type": "commonjs",
  "license": "ISC",
  "dependencies": {
    "canvas": "^2.11.2"
  }
}

Dockerfile:

FROM node:slim

WORKDIR /app

COPY ./package.json ./
RUN npm install

Running docker build --pull --no-cache . fails with

[+] Building 3.3s (8/8) FINISHED                                                                    
 => [internal] load build definition from Dockerfile                                           0.0s
 => => transferring dockerfile: 107B                                                           0.0s
 => [internal] load .dockerignore                                                              0.0s
 => => transferring context: 2B                                                                0.0s
 => [internal] load metadata for docker.io/library/node:slim                                   0.1s
 => [1/4] FROM docker.io/library/node:slim@sha256:bffbb1bf1a3afd6fbdd822bda7f1e7cb07eb407f3dc  0.0s
 => [internal] load build context                                                              0.0s
 => => transferring context: 201B                                                              0.0s
 => CACHED [2/4] WORKDIR /app                                                                  0.0s
 => [3/4] COPY ./package.json ./                                                               0.0s
 => ERROR [4/4] RUN npm install                                                                3.1s
------
 > [4/4] RUN npm install:
#0 3.026 npm notice 
#0 3.026 npm notice New minor version of npm available! 9.6.7 -> 9.7.1
#0 3.026 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.7.1>
#0 3.026 npm notice Run `npm install -g npm@9.7.1` to update!
#0 3.026 npm notice 
#0 3.027 npm ERR! code 126
#0 3.027 npm ERR! path /app/node_modules/canvas
#0 3.028 npm ERR! command failed
#0 3.028 npm ERR! command sh -c node-pre-gyp install --fallback-to-build --update-binary
#0 3.028 npm ERR! sh: 1: node-pre-gyp: Text file busy
#0 3.029 
#0 3.029 npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-06-19T16_22_22_524Z-debug-0.log
------
Dockerfile:6
--------------------
   4 |     
   5 |     COPY ./package.json ./
   6 | >>> RUN npm install
   7 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c npm install" did not complete successfully: exit code: 126

In this example I'm using an canvas package. I have seen this also cause issues e.g. with the @ory/cli package. The issue seems to show up for NPM packages which run some additional scripts during installation. At this point, I'm not sure if this is a Docker issue or NPM issue.

There is some discussion in https://github.com/moby/moby/issues/9547, but neither of proposed solutions (sleep/sync and changing the storage drive to overlay2) are possible here. I can't inject a sleep because the issue happens during an npm install command and I can't change to overlay2 because... I am already using overlay2.

There is some discussion in "dockerfile" text file busy, where the author is advised to revise the files they build and inject sleep. In my case, it happens when working with 3rd party NPM packages, which seem to work fine in other circumstances, so I can't modify that code. I also can't inject sleep because the failure happens during an npm install command.

Output of docker info

Client: Docker Engine - Community
 Version:    24.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 8
  Running: 2
  Paused: 0
  Stopped: 6
 Images: 8
 Server Version: 24.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-43-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 15.3GiB
 Name: mateusz-mcss
 ID: FLNN:HZMZ:5PZG:VHGO:2BCN:6MKS:5C4I:DJQT:DT5J:CIGG:QGFD:UYRS
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false