Here is my problem, visibile in a munin chart:
My used/opened inodes are "suddenly" constantly increasing.
Is there a way to determine which process is keeping the most currently opened inodes? I used the approach of https://unix.stackexchange.com/questions/117093/find-where-inodes-are-being-used and found and cleaned some folders with mails and logs I could get rid of... Still, if the inodes are OPEN, there must be some process keeping them in use, right? It might not necessarily be the folder with the most files where the increase is coming from - or am I wrong there?
So I would like to see who is keeping them open and then track the usage to see where the increase is coming from
Based on Andrew's script, I created a version of it that also shows the process name. As I have some nginx/apache processes running that may respawn, I'd like to see the results on process name.
for dir in /proc/*/fd; do echo -n "$dir "; pid=`expr "$dir" : '\/proc\/\(.*\)\/.*'`; # extract the pid pname=`ps -p $pid -o comm=`; # grab process name echo -n "$pname "; ls $dir 2>/dev/null | wc -l; done | sort -n -k 3
/proc/4612/fd sshd 49 /proc/46470/fd node 60 /proc/5655/fd nginx 66 /proc/6656/fd nginx 76 /proc/7654/fd nginx 81 /proc/8578/fd dovecot 107 /proc/9657/fd nginx 117 /proc/3495/fd java 146 /proc/4785/fd mysqld 382
So the next test would be logging the distribution over time to see what changes and how it correlates with the number of /proc/sys/fs/inode-nr that Morgan mentioned
One year later...
And guess what, end of September is the point where a faulty drive was replaced. So it looks like the whole mess was generated by a disk error. Nevertheless, the scripts still are useful!