dockerlinux-kernelkubernetesout-of-memorygoogle-container-os

How to enable Google Container Optimized OS swap limit support


I'm running Kubernetes/Docker on Google Container Optimized OS on a GCE instance. When I run docker info it says

$ docker info
Containers: 116
 Running: 97
 Paused: 0
 Stopped: 19
Images: 8
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.21+
Operating System: Container-Optimized OS from Google
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.67 GiB
Name: REDACTED
ID: REDACTED
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

The last line says that there is no swap limit support. I'm having trouble figuring out how to enable swap limit support. I found instructions for Ubuntu/Debian here.

My problem is that my docker containers get OOMKilled as soon as they reach their memory limit instead of trying swapping. I want the containers to use swap as a buffer instead of dying immediately.


Solution

  • Container-Optimized OS (COS) is actually configured with swap disabled completely. You can verify this via running cat /proc/meminfo | grep SwapTotal in a COS VM, which will say that it is configured to 0 kB.

    I'm not sure whether it's a good idea to enable swap in your environment, as it may cause more problems (e.g. disk IO starvation/slowdown, kernel hung) if you are using swap frequently.

    But if you wanna try it out, these commands might help you (run all of them as root):

    cos-swap / # sysctl vm.disk_based_swap=1
    vm.disk_based_swap = 1
    cos-swap / # fallocate -l 1G /var/swapfile
    cos-swap / # chmod 600 /var/swapfile
    cos-swap / # mkswap /var/swapfile
    Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
    no label, UUID=406d3dfc-3780-44bf-8add-d19a24fdbbbb
    cos-swap / # swapon /var/swapfile
    cos-swap / # cat /proc/meminfo | grep Swap
    SwapCached:            0 kB
    SwapTotal:       1048572 kB
    SwapFree:        1048572 kB