dockerdevice-mapper

Clean docker environment: devicemapper


I have a docker environment with 2 containers (Jenkins and Nexus, both with their own named volume). I have a daily cron-job which deletes unused containers and images. This is working fine. But the problem is inside my devicemapper:

du -sh /var/lib/docker/
30G docker/

I can each folder in my docker folder: Volumes (big, but that's normal in my case):

/var/lib/docker# du -sh volumes/
14G volumes/

Containers:

/var/lib/docker# du -sh containers/
3.2M    containers/

Images:

/var/lib/docker# du -sh image/
5.8M    image/

Devicemapper:

/var/lib/docker# du -sh devicemapper/
  16G   devicemapper/

/var/lib/docker/devicemapper/mnt is 7.3G /var/lib/docker/devicemapper/devicemapper is 8.1G

Docker info:

Storage Driver: devicemapper
 Pool Name: docker-202:1-xxx-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 5.377 GB
 Data Space Total: 107.4 GB
 Data Space Available: 28.8 GB
 Metadata Space Used: 6.148 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.141 GB
 Udev Sync Supported: true

What is this space and am I able to clean this without breaking stuff?


Solution

  • Don't use a devicemapper loop file for anything serious! Docker has big warnings about this. Devicemapper support has also been deprecated since Docker 18.09, and was removed in Docker 25.0.

    The /var/lib/docker/devicemapper/devicemapper directory contains the sparse loop files that contain all the data that docker mounts. So you would need to use lvm tools to trawl around them and do things. Have a read though the remove issues with devicemapper, they are kinda sorta resolved but maybe not.

    I would move away from devicemapper where possible or use LVM thin pools on anything RHEL based. If you can't change storage drivers, the same procedure will at least clear up any allocated sparse space you can't reclaim.

    Changing the docker storage driver

    Changing storage driver will require dumping your /var/lib/docker directories which contains all your docker data. There are ways to save portions of it but that involves messing around with Docker internals. Better to commit and export any containers or volumes you want to keep and import them after the change. Otherwise you will have a fresh, blank Docker install!

    1. Export data

    2. Stop Docker

    3. Remove /var/lib/docker

    4. Modify your docker startup to use the new storage driver. Set --storage-driver=<name> in /lib/systemd/system/docker.service or /etc/systemd/system/docker.service or /etc/default/docker or /etc/sysconfig/docker

    5. Start Docker

    6. Import Data

    OverlayFS

    The overlay2 driver is the default preferred storage driver and in use in most places. Just use --storage-driver=overlay2

    Direct LVM Thin Pool

    Docker removed support for devicemapper in January of 2024 with the release of v25.0. The official documentation currently favors replacing it with overlay2.

    Instead of the devicemapper loop file you can use an LVM thin pool directly. RHEL makes this easy with a docker-storage-setup utility that distributed with their EPEL docker package. Docker have detailed steps for setting up the volumes manually.

    --storage-driver=devicemapper \
    --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool \
    --storage-opt dm.use_deferred_removal=true
    

    Docker 17.06+ supports managing simple direct-lvm block device setups for you.

    Just don't run out of space in the LVM volume, ever. You end up with an unresponsive Docker daemon that needs to be killed and then LVM resources that are still in use that are hard to clean up.