Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm using the following to count the number of files in a directory, and its subdirectories:

find . -type f | wc -l

But I have half a million files in there, and the count takes a long time.

Is there a faster way to get a count of the number of files, that doesn't involve piping a huge amount of text to something that counts lines? It seems like an inefficient way to do things.

share|improve this question
    
Dupe: stackoverflow.com/questions/1427032/… Doesn't look like there's an ideal solution –  aidan Nov 23 '10 at 10:00
    
ls -1fR | wc -l any faster ? –  Sirex Nov 23 '10 at 10:37
    
What OS are you using? –  ring0 Nov 23 '10 at 13:54
    
On most Unices when counting files like that the bottleneck is in querying the filesystem inode tables. Multiple find commands, or different commands querying the filesystem will generally not run any faster than one. Counting lines of text here is not the slow part, walking the inodes tables is. –  Demosthenex Apr 25 '12 at 19:02
    
Piping is fast enough. The problem here is that reading from the disk is too slow. –  Thorbjørn Ravn Andersen Dec 5 at 12:50

3 Answers 3

up vote 7 down vote accepted

If you have this on a dedicated file-system, or you have a steady number of files overhead, you may be able to get a rough enough count of the number of files by looking at the number of inodes in the file-system via "df -i":

root@dhcp18:~# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            60489728   75885 60413843    1% /

On my test box above I have 75,885 inodes allocated. However, these inodes are not just files, they are also directories. For example:

root@dhcp18:~# mkdir /tmp/foo
root@dhcp18:~# df -i /tmp 
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            60489728   75886 60413842    1% /
root@dhcp18:~# touch /tmp/bar
root@dhcp18:~# df -i /tmp
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            60489728   75887 60413841    1% /

NOTE: Not all file-systems maintain inode counts the same way. ext2/3/4 will all work, however btrfs always reports 0.

If you have to differentiate files from directories, you're going to have to walk the file-system and "stat" each one to see if it's a file, directory, sym-link, etc... The biggest issue here is not the piping of all the text to "wc", but seeking around among all the inodes and directory entries to put that data together.

Other than the inode table as shown by "df -i", there really is no database of how many files there are under a given directory. However, if this information is important to you, you could create and maintain such a database by having your programs increment a number when they create a file in this directory and decrement it when deleted. If you don't control the programs that create them, this isn't an option.

share|improve this answer
    
I was just writing almost exactly the same thing, but you beat me to it, with nice examples and everything. :) One minor addition is that if directories need to be differentiated but don't change often, that number can be cached, or if precision isn't necessary, estimated. –  mattdm Nov 23 '10 at 13:38
    
That's the kind of thing I'm looking for! –  aidan Nov 24 '10 at 18:03

I would also try:

find topDir -maxdepth 3 -printf '%h %f\n'

And then process the output, reducing into a count for the directories.

This is especially useful if you anticipate the directory structure.

share|improve this answer

Try this handy little Python script to see if its any faster.

from os import walk
print sum([len(files) for (root, dirs, files) in walk('/some/path')])

Andrew

share|improve this answer
    
Amazing. But does it actually work? I get only 1/5 of the files compared to find dir -type f | wc -l. –  ring0 Nov 23 '10 at 13:51
    
@ring0: It should be len(files). –  Dennis Williamson Nov 23 '10 at 15:09
    
Oops, for some reason I was thinking dirs. Updated to reflect this. –  Andrew M. Nov 23 '10 at 18:44
    
Seems to take the same amount of time as the find | ls solution. At least, on on this system is does. –  aidan Nov 24 '10 at 18:01
    
Yes, I would imagine it does take the same amount of time as find, it has to do the same number of opendir/readdir/stat operations... –  Sean Reifschneider Nov 27 '10 at 9:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.