etcddefragmentation

Does etcd have an environment DEFRAGMENT_ENABLE?


I have a docker compose that I run on my computer. Data compression works, this is shown in my logs. But for some reason there is run no automatic defragmentation. And that's why I wanted to clarify, whether etcd supports environment DEFRAGMENT_ENABLE, and is it possible to enable automatic defragmentation?

services:
  etcd:
    image: bitnami/etcd:3.5.20
    environment:
      - ALLOW_NONE_AUTHENTICATION=yes
      - ETCD_ADVERTISE_CLIENT_URLS=http://test-etcd:2379
      - ETCD_AUTO_COMPACTION_RETENTION=2
      - ETCD_AUTO_COMPACTION_MODE=periodic
      - DEFRAGMENT_ENABLE=true
      - DEFRAGMENT_PERIODIC_INTERVAL=3

Below is the etcd log

{"level":"info","ts":"2025-03-22T01:23:46.712668Z","caller":"mvcc/hash.go:151","msg":"storing new hash","hash":2822278584,"revision":9,"compact-revision":-1}
{"level":"info","ts":"2025-03-22T02:23:46.717065Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T02:23:46.739152Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"22.038708ms"}
{"level":"info","ts":"2025-03-22T02:23:46.739185Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":37010}
{"level":"info","ts":"2025-03-22T02:23:47.218714Z","caller":"mvcc/kvstore_compaction.go:71","msg":"finished scheduled compaction","compact-revision":37010,"took":"479.103958ms","hash":1316096291,"current-db-size-bytes":6094848,"current-db-size":"6.1 MB","current-db-size-in-use-bytes":184320,"current-db-size-in-use":"184 kB"}
{"level":"info","ts":"2025-03-22T02:23:47.218750Z","caller":"mvcc/hash.go:151","msg":"storing new hash","hash":1316096291,"revision":37010,"compact-revision":9}
{"level":"info","ts":"2025-03-22T03:23:46.745119Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T03:23:46.753841Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"8.662593ms"}
{"level":"info","ts":"2025-03-22T04:23:46.760425Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T04:23:46.769094Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"8.626125ms"}
{"level":"info","ts":"2025-03-22T05:23:46.776632Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T05:23:46.785284Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"8.615024ms"}
{"level":"info","ts":"2025-03-22T06:23:46.791949Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T06:23:46.800617Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"8.630073ms"}
{"level":"info","ts":"2025-03-22T07:23:46.805638Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T07:23:46.814255Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"8.576452ms"}
{"level":"info","ts":"2025-03-22T08:23:46.820041Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T08:23:46.828711Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"8.627478ms"}
{"level":"info","ts":"2025-03-22T09:23:46.833890Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T09:23:46.846081Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"12.145451ms"}
{"level":"info","ts":"2025-03-22T10:23:46.851789Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T10:23:46.852438Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"611.259µs"}
{"level":"info","ts":"2025-03-22T11:23:46.857700Z","caller":"v3compactor/periodic.go:134","msg":"starting auto periodic compaction","revision":37010,"compact-period":"2h0m0s"}
{"level":"info","ts":"2025-03-22T11:23:46.867077Z","caller":"v3compactor/periodic.go:142","msg":"completed auto periodic compaction","revision":37010,"compact-period":"2h0m0s","took":"9.333535ms"}

Solution

  • TL;DR

    No, bitnami/etcd currently (v3.6) doesn't provide a flag for defragmentation.

    Long Answer

    According to bitnami/etcd image README, there is no such flag as ETCD_DEFRAGMENT_*, nor anything similar.

    If you want to perform automatic defragmentation, unless Bitnami integrates it in their Docker images, you can:

    For reference, Etcd defragmentation is documented here:

    Note that combining defragmentation with auto compaction is a great practice, but I see you already enabled the latter :)

    1. Sidecar

    We can create a sidecar container, under the same virtual network of the original etcd container, whose only purpose is to run etcdctl defrag to perform defragmentation on the main etcd container.

    Note that this solution isn't optimal:

    For more control, you would have to use an orchestrating framework such as Kubernetes, which provides CronJob abstraction (better solution) or a custom Docker image.

    Example (Docker Compose)

    The following example shows how to setup a sidecar container to run defragmentation once every 10 seconds.

    File compose.yaml:

    name: defrag-example
    
    services:
      etcd:
        image: bitnami/etcd:3.5.20
        environment:
          ALLOW_NONE_AUTHENTICATION: yes
        ports:
          - "2379:2379"
        healthcheck:
          test: "etcdctl endpoint health"
          interval: 5s
          timeout: 30s
          retries: 5
        networks:
          etcd-net:
    
      # Sidecar container for defragmentation
      etcd-defragger:
        image: bitnami/etcd:3.5.20
        environment:
          ALLOW_NONE_AUTHENTICATION: yes
        restart: always
        depends_on:
          etcd:
            condition: service_healthy
        entrypoint: |
          sh -c 'echo "Starting defragmentation at $(date)"; etcdctl defrag --endpoints=http://etcd:2379; sleep 10s'
        networks:
          etcd-net:
    
    networks:
      etcd-net:
        driver: bridge
    

    Run:

    docker compose up
    

    Result:

    [...]
    etcd-defragger-1  | Starting defragmentation at Tue Jun 10 08:26:05 UTC 2025
    etcd-1            | {"level":"info","ts":"2025-06-08T19:26:06.005422Z","caller":"v3rpc/maintenance.go:92","msg":"starting defragment"}
    etcd-1            | {"level":"info","ts":"2025-06-08T19:26:06.013405Z","caller":"backend/backend.go:517","msg":"defragmenting","path":"/bitnami/etcd/data/member/snap/db","current-db-size-bytes":20480,"current-db-size":"20 kB","current-db-size-in-use-bytes":16384,"current-db-size-in-use":"16 kB"}
    etcd-1            | {"level":"info","ts":"2025-06-08T19:26:06.030905Z","caller":"backend/backend.go:579","msg":"finished defragmenting directory","path":"/bitnami/etcd/data/member/snap/db","current-db-size-bytes-diff":0,"current-db-size-bytes":20480,"current-db-size":"20 kB","current-db-size-in-use-bytes-diff":-4096,"current-db-size-in-use-bytes":12288,"current-db-size-in-use":"12 kB","took":"25.385613ms"}
    etcd-1            | {"level":"info","ts":"2025-06-08T19:26:06.030976Z","caller":"v3rpc/maintenance.go:100","msg":"finished defragment"}
    etcd-1            | {"level":"info","ts":"2025-06-08T19:26:06.030997Z","caller":"v3rpc/health.go:61","msg":"grpc service status changed","service":"","status":"SERVING"}
    etcd-defragger-1  | Finished defragmenting etcd member[http://etcd:2379]
    etcd-defragger-1 exited with code 0
    

    2. Custom Docker Image

    We have to:

    While trying to implement this one I stumbled upon some issues, such as the fact that crontab doesn't allow to create crontabs to users not present in /etc/passwd (such as default user with UID 1001). But I've not investigated further.

    If I resolve these issues, I’ll update the answer.

    3. Kubernetes

    Kubernetes has native support for scheduled containers, using CronJob resource (see Kuberentes Docs | CronJob and Kubernetes API | CronJob v1).

    Example ("Raw" Resource)

    Kubernetes resource etcd-defrag-cronjob.yaml:

    apiVersion: batch/v1
    kind: CronJob
    metadata:
      name: etcd-defragger
    spec:
      concurrencyPolicy: Forbid
      schedule: "0 0 * * *"
      suspend: false
      jobTemplate:
        spec:
          template:
            spec:
              containers:
                - name: etcd-defrag
                  image: docker.io/bitnami/etcd:bitnami/etcd:latest
                  imagePullPolicy: "IfNotPresent"
                  command:
                    - "etcdctl"
                  args:
                    - defrag
                    # Here you specify the Etcd endpoint (typically you would point to the Service hostname)
                    - --endpoints=[]
    

    Install with:

    kubectl apply etcd-defrag-cronjob.yaml
    

    Example (Bitnami Helm Chart)

    bitnami/etcd Helm chart supports both auto compaction (based on flag ETCD_AUTO_COMPACTION_*) and defragmentation. The latter is implemented through a CronJob template.

    The following example configure the defragmentation cron job to run every minute. File values.yaml:

    autoCompactionMode: periodic
    autoCompactionRetention: 24h
    
    defrag:
      enabled: true
      cronjob:
        schedule: "* * * * *"
    

    Install with:

    helm install etcd-test oci://registry-1.docker.io/bitnamicharts/etcd -f values.yaml
    

    While the chart is installed, we can see that the CronJob spawns a Job and a Pod, which runs the defragmentation:

    $ kubectl get jobs
    NAME                        STATUS     COMPLETIONS   DURATION   AGE
    etcd-test-defrag-29159875   Complete   1/1           5s         11s
    
    $ kubectl logs etcd-test-defrag-29159875-vkpjx
    Finished defragmenting etcd member[http://etcd-test-0.etcd-test-headless.default.svc.cluster.local:2379]. took 139.565377ms
    

    And watching the logs of the main Etcd pod, we can see that defragmentation was performed:

    $ kubectl logs -f etcd-test-0
    ...
    {"level":"info","ts":"2025-06-10T21:57:01.823712Z","caller":"v3rpc/maintenance.go:110","msg":"starting defragment"}
    {"level":"info","ts":"2025-06-10T21:57:01.835577Z","caller":"backend/backend.go:522","msg":"defragmenting","path":"/bitnami/etcd/data/member/snap/db","current-db-size-bytes":20480,"current-db-size":"20 kB","current-db-size-in-use-bytes":12288,"current-db-size-in-use":"12 kB"}
    {"level":"info","ts":"2025-06-10T21:57:01.859304Z","logger":"bbolt","caller":"backend/backend.go:574","msg":"Opening db file (/bitnami/etcd/data/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: hashmap, ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00040e198}"}
    {"level":"info","ts":"2025-06-10T21:57:01.859597Z","logger":"bbolt","caller":"bbolt@v1.4.0/db.go:321","msg":"Opening bbolt db (/bitnami/etcd/data/member/snap/db) successfully"}
    {"level":"info","ts":"2025-06-10T21:57:01.859663Z","caller":"backend/backend.go:592","msg":"finished defragmenting directory","path":"/bitnami/etcd/data/member/snap/db","current-db-size-bytes-diff":0,"current-db-size-bytes":20480,"current-db-size":"20 kB","current-db-size-in-use-bytes-diff":0,"current-db-size-in-use-bytes":12288,"current-db-size-in-use":"12 kB","took":"35.84145ms"}
    {"level":"info","ts":"2025-06-10T21:57:01.859711Z","caller":"v3rpc/maintenance.go:118","msg":"finished defragment"}
    ...
    

    To uninstall the Helm chart:

    helm uninstall etcd-test
    

    To see the generated resources you can run the following:

    helm fetch oci://registry-1.docker.io/bitnamicharts/etcd --untar
    helm template ./etcd/ --dry-run -f values.yaml > resources.yaml
    

    And see how the CronJob is actually implemented.

    Side note: I contributed to the defragmentation CronJob template on Bitnami's Helm chart :)