I have a single server running some code that creates millions of small files all in a single directory. At first, the code runs extremely fast. After a day or so, it begins to slow down quite a bit. I'm trying to figure out whether this is an issue with the code, or whether this is an issue with the number of files in the directory.

The code itself creates a file if one does not exist, then opens the file and writes a single line to it. If the file does exist, it will open the file and append a single line to it.

With that said, I've been benchmarking my code and making changes as need be, but I'd like to benchmark XFS as well to see if that is the bottleneck. My question is, how do I benchmark the read/write performance of XFS?

  • What does lsof say? This seems likely to be an issue with file handles unless you are trying to do the reads/writes simultaneously. – TheFiddlerWins Aug 25 '17 at 12:06
  • millions of small files all in a single directory That's going to be slow on any type of filesystem, and absolutely abysmal on some. – Andrew Henle Aug 27 '17 at 12:15

Start with your application and understand how it performs.

  • Add instrumentation to your code so you can measure how long tasks take.
  • Monitor host OS metrics like CPU and storage IOPS.
  • File system and block device response time with dynamic tracing like xfsslower and bioslower
  • Capture profiles with tools like perf record so you can see what is hot on CPU with flame graph visualizations. Try it with debug symbols for kernel and user programs.

Millions of files in one directory is usually not a good idea. Even if it is supported, you get giant directory metadata that take a long time to list. Consider your application making a directory structure that limits files per directory to a few thousand.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.