inotify problemThis time, we do troubleshooting inotify issue: too many open files. the story begun when there were many users having difficulties of accessing VM’s console on proxmox hypervisor. when users clicked the console menu, it doesnt appeared as expected, only gives a blank pages with error message like the picture above. Some of can access smoothly, but others failed.

By observing this behaviour, it seems like there an issue with resource limitation where when it reached the threshold level more users cannot be served properly.

Symptoms:

when i check the error log, i got message

Unable to create new inotify object: Too many open files at /usr/share/perl5/PVE/INotify.pm line 389.
TASK ERROR: Failed to run vncproxy

and when i check the line 389 on the file, i got the picture below.

inotify code

well, it looks like the problem is related to linux inotify features where other people also experiencing the same. inotify is a kernel parameter which can be adjusted through sysctl command.

so i run “sysctl -a –pattern inotify” to find out which parametes related to inotify and found below:

fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192

I then increases each original value above by 4 folds and save the entries to /etc/sysctl.conf file in order to make it permanent.

the result, access to container and console is now better 🙂

note: this inotify issue also happens when you have lots of virtual machines or containers. the setting above could be a relieve. 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.