AWS EC2 terminal session terminated with "Plugin with name Standard_Stream not found"

I was streaming Kafka on AWS EC2 CentOS 7. My Session Manager Idle Timeout is set to 60min. And yet, after running for much less than that, the terminal got frozen, saying My session has been terminated. Of course, the Kafka streaming for disrupted as well.

When I tried to restart a new session with a new terminal, I got this error popup

Your session has been terminated for the following reasons: Plugin with name Standard_Stream not found. Step name: Standard_Stream

and I am still unable to restart a terminal.

What does this error mean and how to resolve it? Thanks.

SSM agent can report this error in two situations.

1. Total number of open files across the system has reached system wide limit of max open files.
2. System has reached limit for inotify sub system in the kernel

Increase the max open files:

$ sudo sysctl -w fs.file-max=100000

To set it permanent, append below line to /etc/sysctl.conf

$ fs.file-max = 100000

Checking the limits on inotify subsystem:

$ sudo cat /proc/sys/fs/inotify/max_user_instances: 

This specifies an upper limit on the number of inotify instances that can be created per real user ID. Default is 128

$ sudo cat /proc/sys/fs/inotify/max_user_watches:

This specifies an upper limit on the number of watches that can be created per real user ID. Default is 8192

To check the inotify resource consumption per process, you can run below command.

$ sudo find /proc/*/fd -lname anon_inode:inotify | cut -d/ -f3 | xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' | uniq -c | sort -nr

Test with tail -f to see if there is still room for new inotify watchers. Run this as root user and look for any message similar to "tail: inotify cannot be used, reverting to polling: Too many open files"

$ sudo su -

$ tail -f /var/log/syslog

If you get that error, it is then that the system has reached one of above limits for inotify subsystem. Increase the limits or reboot to clear. Test it again using tail -f to see if new watchers are being created after the changes.

$ sudo sysctl fs.inotify.max_user_watches=16384

$ sudo sysctl fs.inotify.max_user_instances=256

To make this change persistent across reboot.

$ echo "fs.inotify.max_user_watches = 16384” >> /etc/sysctl.d/20-fs-inotify.conf

$ echo "fs.inotify.max_user_instances = 256” >> /etc/sysctl.d/20-fs-inotify.conf