[SOLVED] "no space left on device" when building with Jenkins and Docker

"no space left on device" when building with Jenkins and Docker

we use Jenkins to build Docker images and everything was find until last week when every time I try to build an image in Jenkins it fails with "Error response from daemon: Error processing tar file(exit status 1): write /app/node_modules/acorn/dist/acorn_loose.es.js: no space left on device" (the file it fails on can depend on the project). The image will build if I try using Docker in the server but fails on Jenkins.

I have tried removing old containers and images etc but to no avail. Disk space and inodes seem to be fine so I'm not sure what to try now. Any help is appreciated.

Result of "docker info":

Containers: 55
 Running: 48
 Paused: 0
 Stopped: 7
Images: 59
Server Version: 17.03.2-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.45-rancher
Operating System: RancherOS v1.1.0
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 19.61 GiB
Name: rancher
ID: Z7Z3:T3NW:N4O3:FKMZ:7KH6:FJ7R:TJ6A:FXLW:KNUL:WMRC:ED74:KHEM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 dockerhub.companysite.net:5000
 127.0.0.0/8
Live Restore Enabled: false

Result of "df -h":

Filesystem      Size  Used Avail Use% Mounted on
overlay          47G   27G   18G  60% /
tmpfs           9.9G     0  9.9G   0% /dev
tmpfs           9.9G     0  9.9G   0% /sys/fs/cgroup
/dev/sda1        47G   27G   18G  60% /.r
shm              64M     0   64M   0% /dev/shm
tmpfs           9.9G     0  9.9G   0% /sys/firmware

Result of "df -ih":

Filesystem     Inodes IUsed IFree IUse% Mounted on
overlay           13M  2.8M  9.8M   22% /
tmpfs            2.5M    16  2.5M    1% /dev
tmpfs            2.5M    15  2.5M    1% /sys/fs/cgroup
/dev/sda1         13M  2.8M  9.8M   22% /.r
shm              2.5M     1  2.5M    1% /dev/shm
tmpfs            2.5M     1  2.5M    1% /sys/firmware

Solution

You may have a bunch of old dangling images left over that are causing problems, try:

docker volume rm $(docker volume ls -qf dangling=true)

(This does delete stuff, so you may want to check there is nothing you want to keep in any dangling volumes before doing this)