3

I have a NFS directory mounted on a host. That directory has 0.6 million log files now, and will have 1.6 million eventually. The files are small, most of them are less than 1MB.

The problem is that I could not reliably find all files of a day in that directory.

If I run such a command below, I should get 4320 files for a day, but I could get any number from 1 to 4320, for example:

$ find /mnt/log -type f -name "some-prefix-rolling.log.2015-07-05*" | wc -l
2548

I have to read this directory as it is. I could not make any change, e.g. put one day's log files in one folder, because some other applications depend on this setting.

The mount options are: ro,noatime,bg,hard,rsize=32768,wsize=32768,vers=3

Does anybody know how to fix this issue?

Improve this question
3
  • 2
    Very few filesystems can handle such a large number of files in a single directory without severe performance problems. Split them into multiple directories.
    – Michael Hampton
    Jul 20, 2015 at 18:44
  • I would agree with @MichaelHampton, trying to find a way to make the application that relies on files in that directory to search a hierarchy would be a good option. Alternately would it be possible to zip/compress a days files using gzip or something similar, then add an extra step before the application runs that unpacks the appropriate day(s) logs from the .gz? It seems like a hassle, but if creating subdirectories is not an option packing and unpacking the files into a zipped format might be the best option.
    – Matt
    Jul 20, 2015 at 19:02
  • @MichaelHampton: while it would be ideal to use subdirs, modern filesystem as XFS and ZFS should have no problem in managing millions of file per directory. Non up-to-date userspace tools can have problem, though.
    – shodanshok
    Jul 20, 2015 at 19:03

2 Answers 2

Reset to default
2

While storing files in subdirs would be ideal, the right (and expected) behavior is not what you are seeing. Some hints to track down the problem:

  • check your source filesystem: if you run your command directly on the data source, does it complete correctly?
  • for so many files, your source filesystem should be XFS or ZFS. Avoid EXT4 and BTRFS
  • try to toggle client-side caching (FS-cache module)
  • does a simple ls -al | wc -l return consistent results?
Improve this answer
1

This post gives some of the upper limits for filesystems. Regardless of the "hard limit" of how many files can be stored in a directory, you will very quickly run into performance issues...In the 10's of thousands of files, not the 100's of thousands, especially if you are using a utility like find which has to parse the entire directory.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.