non-cpu-intensive alternative to lsof?

Question

We run an Apache Cassandra cluster where each host has a few hundred thousand files open at any given time.

We'd like to be able to get a count of open files at periodic intervals and feed this number into graphite, but when we run lsof under collectd, it ends up taking a few minutes to complete and chewing up an inordinate amount of CPU in the meantime.

I'm wondering if there's an alternate and more friendly means of getting the same data that lsof provides, or even a way of running lsof that won't eat into CPU as noticeably? (Although I assume this latter method would likely take much longer to complete than it currently does... not ideal).

Perhaps the kernel maintains some variable somewhere that contains the number of open files? Wishful thinking?

Update:

In reponse to one of the answers, we're already using the -b and -n flags. Here's the full command as I have it running under collectd:

sudo lsof -b -n -w | stdbuf -i0 -o0 -e0 wc -l

Answer 1

You probably don't need to resolve the network addresses for socket, so a least use the -n switch. Then you may also want so skip blocking operations with -b.

These 2 first switches should really make it faster.

And then -l to avoid resolving uids. And -L to avoid counting links. Etc. See the man lsof .

Alternatively, with Linux, you could make a script to simply count the links under /proc/<PID>/fd like this:

find /proc -mindepth 3 -maxdepth 3 -type l | awk -F/ '$4 == "fd" { s++ } END { print s }'

Answer 2

You're doing it wrong.

From man proc

   /proc/sys/fs/file-nr

This (read-only) file contains three numbers: the number of allocated file handles (i.e., the number of files presently opened); the number of free file handles; and the maximum number of file handles (i.e., the same value as /proc/sys/fs/file-max). If the number of allocated file handles is close to the maximum, you should consider increasing the maximum. Before Linux 2.6, the kernel allocated file handles dynamically, but it didn't free them again. Instead the free file handles were kept in a list for reallocation; the "free file handles" value indicates the size of that list. A large number of free file handles indicates that there was a past peak in the usage of open file handles. Since Linux 2.6, the kernel does deallocate freed file handles, and the "free file handles" value is always zero.

The first value if you cat that gives you precisely what you are after it would appear.

For the record I couldn't get the lsof output to match it even with some amount of fudging but I gather if thats what the kernel says its more authoritative than the list you get from lsof anyway.