Am getting the "no space left on device" error while removing a container.
# docker rm orientdb1
Error response from daemon: Driver devicemapper failed to remove root filesystem ba18ea8014b6024eb738502612130244ff363b433f146102c64eb2eb0f3a321c: devmapper: Error saving transaction metadata: devmapper: Error creating metadata file: open /var/lib/docker/devicemapper/metadata/.tmp636968871: no space left on device
Here is a snippet from "df -kh" output.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 3.8T 3.8T 20K 100% /
Here is a snippet from "du -ksh " output. Which shows that 3.8TB is used.
3.8T /var/lib/docker
This is the "docker info" output.
# docker info
Containers: 56
Running: 8
Paused: 0
Stopped: 48
Images: 18
Server Version: 1.12.0
Storage Driver: devicemapper
Pool Name: docker-253:0-1196768-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 24.51 GB
Data Space Total: 107.4 GB
Data Space Available: 20.48 kB
Metadata Space Used: 29.18 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 20.48 kB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for
production use. Use `--storage-opt dm.thinpooldev` to specify a custom
block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host overlay null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 251.7 GiB
Name: my-host1
ID: KTTY:PSTQ:YQSP:9KZJ:VSVU:DFUZ:CEMF:DYWQ:CA3I:6KUB:6J2J:D4NG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Would like to understand how 3.8 TB is accounted for from the following (snippet from "docker info")
Data Space Used: 24.51 GB
Data Space Total: 107.4 GB
Data Space Available: 20.48 kB
Metadata Space Used: 29.18 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 20.48 kB
Thin Pool Minimum Free Space: 10.74 GB
Also how can I clean /var/lib/docker to get some free space.
I believe the disk space figures are for the loopback driver and the file /var/lib/docker/devicemapper/devicemapper/data that is used by devicemapper in the loopback mode.
To cleanup the docker, you can run docker system prune on a newer release. Older releases can use commands like:
docker ps -aqf status=exited | xargs docker rm
docker images -qf dangling=true | xargs docker rmi
Beyond that, you will need to go through containers and images to docker stop, docker rm on the container, and docker rmi on the image.