3

I'm running find . -type d on a rather large directory tree. I am only interested in finding directories within this tree, but when I ran an strace against the process to make sure it was doing what I expected it to be doing, I noticed that there are a huge amount of operations being wasted running fstat against files within the tree.

newfstatat(AT_FDCWD, "file1", {st_mode=S_IFREG|0600, st_size=7690, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "file2", {st_mode=S_IFREG|0600, st_size=7696, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "file3", {st_mode=S_IFREG|0600, st_size=7687, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "file4", {st_mode=S_IFREG|0600, st_size=10455, ...}, AT_SYMLINK_NOFOLLOW) = 0

Is find not aware that an inode is pointing to a directory until it performs an fstat? If that's the case, then this is going to take a long time. Some of these directories likely have millions of items within them, but I really only care about directories.

Ultimately I would like a report of the dirsize and path of each of the directories in my file tree. What's the fastest/most efficient way for me to do that?

  • Directories are infact files. 'find' uses fstat to find out if a file is - a file, a special file or a directory. From what I understand, even other core-utils tools use fstat to determine the basic information about a file. I don't think there is any more efficient way to read inode metadata other than stat syscall. – vagarwal Oct 25 '14 at 6:31
  • Hmmm....isn't the proper command "find . -type d -print"?? – mdpc Oct 25 '14 at 6:40
  • 1
    -print is an implicit outcome in modern find. In fact, it's even worse: everything can be implied: find alone implies find . -print. – MadHatter Oct 25 '14 at 7:12
  • @mdpc correct the command as posted gives an error but also note that -print is implicit on modern find implementations. – user9517 Oct 25 '14 at 7:13
  • The actual command I am running is find */. -type d -exec ls -ld {} +, so that I can see the size/location of the directory. Sorry for the confusion! – Nathan Oct 25 '14 at 14:14
4

Yes, it looks like it really is the case that find is using fstat to determine the type of the file. This is mildly surprising given that dirent has contained the information since kernel 2.6.4.

Not all filesystems have support for the extended dirent behaviour so either this is true in your case or find doesn't use it. Without knowing your filesystem type we can't decide.

  • It's an NFS mount. Interestingly, I ran a perl script using IO::Dirent and it seems to be able to enumerate directories much faster, so perhaps find isn't the best tool for the job. – Nathan Oct 26 '14 at 1:27
  • Interestingly there seems to be a feature of find called d_type optimisation which can be enabled with -o2 but it doesn't seem to have much effect in my case :( – Nathan Oct 26 '14 at 1:53
1

As I am sure you know, a directory is a special type of file in the UNIX paradigm. To determine whether something is a directory or another type of file, it must be interrogated, and fstat() is a good way to do this.

I believe later filesystems and fs-drivers keep a separate table of just the directories, but the find command dates back decades, and is probably either not adapted to newer filesystems or it maintains downward compatibility.

You could fake this by running a recurring job out of CRON (at a nice value >0 if you want to be easy on the IO utilization for other processes) that does a:

find ${DIRECTORY} -type d -print >${DIRECTORY}/.only_folders

Then when you need this, use the contents of the file that you prebuilt instead of traversing the directory again.

cat "${DIRECTORY}/.only_folders" |while read FOLDER ; do
  do_work.sh ${FOLDER} ;
done

instead of something like

find ${DIRECTORY} -type d |xargs do_work.sh

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.