[SOLVED] Drone sometimes fail with error related to "No space left on device"

Drone sometimes fail with error related to "No space left on device"

Sometimes we experience a drone pipeline failing due to a lack of disk space, but there is a lot of space.

drone@drone2:~$ df -H
Filesystem      Size  Used Avail Use% Mounted on
udev            8.4G     0  8.4G   0% /dev
tmpfs           1.7G  984k  1.7G   1% /run
/dev/vda2       138G   15G  118G  12% /
tmpfs           8.4G     0  8.4G   0% /dev/shm
tmpfs           5.3M     0  5.3M   0% /run/lock
tmpfs           8.4G     0  8.4G   0% /sys/fs/cgroup
overlay         138G   15G  118G  12% /var/lib/docker/overlay2/cf15a2d5e215d6624d9239d28c34be2aa4a856485a6ecafb16b38d1480531dfc/merged
overlay         138G   15G  118G  12% /var/lib/docker/overlay2/c4b441b59cba0ded4b82c94ab5f5658b7d8015d6e84bf8c91b86c2a2404f81b2/merged
tmpfs           1.7G     0  1.7G   0% /run/user/1000

We use docker images for running drone infostracture. The console command:

docker run \
  --volume=/var/lib/drone:/data \
  --publish=80:80 \
  --publish=443:443 \
  --restart=always \
  --detach=true \
  --name=drone \
  <drone image>

My assumption may be due to the docker container's limitations, and we need to configure it somehow manually.

Any suggestion on how to fix it?

Solution

We have manage to identify the problem. Our cloud provider has confirmed that our software has used the disc a lot this lead to triggering limitations on disk operation. They gave us a recommendation to increase the disk plan. Or optimise the disk usage.