3

We're seeing performance problems on a file system that contains ~500 M files that are relatively small (typically tens of kb) and ~150 K directories. Files access is predominantly reads and writes are fairly rare. Files are mostly stored in a hashed directory hierarchy with ~30K files per directory, but in some cases there can be up to 500 K files in one directory.

The server shares the file system to ~10 client machines.

Getting directory listings is often slow and sometimes also reading files by absolute paths, even locally.

This is a physical server that runs Ubuntu 12.04.4 (kernel: 3.8.0 / x86_64) and we use ext4 on a hardware RAID-6 volume.

What would be a recommended file system setup in this scenario?

  • is there a file system particularly well suited for this case (e.g. ext4, xfs, btrfs, ...)?
  • what kind of RAID configuration (e.g. sw vs. hw, RAID level etc.) should we use?
  • what about the file sharing setup (technology [e.g. NFS vs. competition], configuration, ...)?
  • If it's just a file sharing, I'd go with FreeBSD/ZFS, or something Illumos based and a huge amount of memory. Maybe SSD for L2ARC. – Fox Jun 26 '15 at 8:08
  • How much memory do you have there? How free -h looks like after some heavy usage? – Michal Sokolowski Jun 26 '15 at 11:00
  • the system has 32 GB of memory. – marko Jun 26 '15 at 20:08
  • Protip: reiserfs for small files – dmourati Jun 28 '15 at 19:10
2

When you have a problem like this, you have to:

  • obtain all requirements (latency, bandwidth, redundancy, reliability, security, required features...)
  • analyse the current systems. If they are none, create test environments. Understand how all components work. Understand the current and the expected load.
  • add system monitoring (with graphs) for both production and test systems. Monitor at least CPU usage, network usage and Disk I/O usage.
  • create test servers and load test them. Load test with synthetic benchmarks not only with micro-benchmarks.

Use stable versions of recent OS with latest stable kernel.

0

What would be a recommended file system setup in this scenario?

is there a file system particularly well suited for this case (e.g. ext4, xfs, btrfs, ...)?

I think ext4 is the best choice here in terms of safety. In terms of reading performance RaiserFS(/4) could be better:

https://unix.stackexchange.com/questions/28756/what-is-the-most-high-performance-linux-filesystem-for-storing-a-lot-of-small-fi

http://www.phoronix.com/scan.php?page=article&item=linux_39_fs&num=1

filesystem for millions of small files

what kind of RAID configuration (e.g. sw vs. hw, RAID level etc.) should we use?

You need to:

  • shorten latency / access time
  • reduce number of reads/writes (noatime,nodiratime,realtime,barrier=0)

Every RAID level, which has possibility to do parallel operations will be good choice, for example RAID 10 (Linux software raid) has a lot tuning options.

what about the file sharing setup (technology [e.g. NFS vs. competition], configuration, ...)?

I can't give you numbers here, file sharing is also quite wide term. I've had very good experience with FTP when dealing with lots of files (not FTPS - because negotiating TLS for every transfer takes time), better tunnel the traffic. It seems that the best division between flexibility of the protocol (what u can do) and protocol overhead (how much time it takes, space in the packet) when coping small files.

  • We'd basically need a distributed file system akin to NFS, but we're open other options if something else would be a better fit in our case. – marko Jun 26 '15 at 20:13
  • Sorry, but it means nothing. What do you need? Block device sharing support, extended ACL's, xattr, compression? It sounds like you didn't analyze it yet. – Michal Sokolowski Jun 26 '15 at 20:49
  • We need to share files across multiple systems. Currently, there's only one system that changes the filesystem at a given time and the filesystem contents need to be readable from two other systems. Compression or ACLs are not needed. – marko Jul 2 '15 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.