1

I have a single server running some code that creates millions of small files all in a single directory. At first, the code runs extremely fast. After a day or so, it begins to slow down quite a bit. I'm trying to figure out whether this is an issue with the code, or whether this is an issue with the number of files in the directory.

The code itself creates a file if one does not exist, then opens the file and writes a single line to it. If the file does exist, it will open the file and append a single line to it.

With that said, I've been benchmarking my code and making changes as need be, but I'd like to benchmark XFS as well to see if that is the bottleneck. My question is, how do I benchmark the read/write performance of XFS?

Improve this question
3
  • What does lsof say? This seems likely to be an issue with file handles unless you are trying to do the reads/writes simultaneously.
    – TheFiddlerWins
    Commented Aug 25, 2017 at 12:06
  • millions of small files all in a single directory That's going to be slow on any type of filesystem, and absolutely abysmal on some.
    – Andrew Henle
    Commented Aug 27, 2017 at 12:15
  • I'm seeing similar behavior as to the poster. Initially copying at 100mb/s however once I get to about 500GB of millions of small files, it gets really really slow like 10MB/s.... Probably something fundamentally to how XFS works.
    – Chris Stryczynski
    Commented Jun 29, 2023 at 15:34

1 Answer 1

Reset to default
1

Start with your application and understand how it performs.

  • Add instrumentation to your code so you can measure how long tasks take.
  • Monitor host OS metrics like CPU and storage IOPS.
  • File system and block device response time with dynamic tracing like xfsslower and bioslower
  • Capture profiles with tools like perf record so you can see what is hot on CPU with flame graph visualizations. Try it with debug symbols for kernel and user programs.

Millions of files in one directory is usually not a good idea. Even if it is supported, you get giant directory metadata that take a long time to list. Consider your application making a directory structure that limits files per directory to a few thousand.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .