lsyncd - OVERFLOW in event queue - Solution is to tune fs.inotify.max_queued_events

lsyncd is a fantastic alternative to NFS or NAS for replicating files among your Linux hosts. I have found the daemon works well with large Linux filesystems (many files, small to large sizes, xfs, ext4, luks) but requires some sysctl tuning as your filesystem grows.

This "question" is a note to myself so I can always find the answer via searching on stack overflow. Hope it helps you!

Github Project: https://github.com/axkibe/lsyncd

Exception in /var/log/lsyncd.log:

Thu Jun 18 17:48:52 2020 Normal: --- OVERFLOW in event queue ---
Thu Jun 18 17:48:52 2020 Normal: --- HUP signal, resetting ---
Thu Jun 18 17:48:52 2020 Normal: waiting for 1 more child processes.

Solution

Solution when you see "OVERFLOW in event queue" in lsyncd.log

From other knowledge bases, I had learned to tune max_user_watches, but by also tuning the max_queued_events, I corrected an OVERFLOW in event queue exception.

The temporary solution worked without needing to restart my lsyncd process.

I picked the number 1000000 as an arbitrarily large number. The default Ubuntu 18 value is 16384.

Temporary Solution

Check your current tuning values:

$ sysctl fs.inotify.max_queued_events
fs.inotify.max_queued_events = 16384
$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

Update both max_user_watches and max_queued_events via shell

sudo sysctl fs.inotify.max_user_watches=1000000
sudo sysctl fs.inotify.max_queued_events=1000000

Permanent Solution, Persists after reboot

Update both max_user_watches and max_queued_events in /etc/sysctl.conf

fs.inotify.max_user_watches=1000000
fs.inotify.max_queued_events=1000000

Lsyncd.conf Basic Configuration

/etc/lsyncd/lsyncd.conf

settings {
    logfile = "/var/log/lsyncd.log",
    pidfile = "/var/run/lsyncd/lsyncd.pid",
    insist = true
}
sync {
   default.rsyncssh,
   source="/var/application/data",
   host="node2",
   excludeFrom="/etc/lsyncd/exclude",
   targetdir="/var/application/data",
   rsync = {
     archive = true,
     compress = false,
     whole_file = true
   },
   ssh = {
     port = 22
   }
}

System Details

Linux service1staging 5.0.0-36-generic #39~18.04.1-Ubuntu SMP Tue Nov 12 11:09:50 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Ubuntu 18.04.4 LTS

lsyncd --version Version: 2.1.6