3

We're seeing performance problems on a file system that contains ~500 M files that are relatively small (typically tens of kb) and ~150 K directories. Files access is predominantly reads and writes are fairly rare. Files are mostly stored in a hashed directory hierarchy with ~30K files per directory, but in some cases there can be up to 500 K files in one directory.

The server shares the file system to ~10 client machines.

Getting directory listings is often slow and sometimes also reading files by absolute paths, even locally.

This is a physical server that runs Ubuntu 12.04.4 (kernel: 3.8.0 / x86_64) and we use ext4 on a hardware RAID-6 volume.

What would be a recommended file system setup in this scenario?

  • is there a file system particularly well suited for this case (e.g. ext4, xfs, btrfs, ...)?
  • what kind of RAID configuration (e.g. sw vs. hw, RAID level etc.) should we use?
  • what about the file sharing setup (technology [e.g. NFS vs. competition], configuration, ...)?
4
  • If it's just a file sharing, I'd go with FreeBSD/ZFS, or something Illumos based and a huge amount of memory. Maybe SSD for L2ARC.
    – Fox
    Jun 26, 2015 at 8:08
  • How much memory do you have there? How free -h looks like after some heavy usage? Jun 26, 2015 at 11:00
  • the system has 32 GB of memory.
    – marko
    Jun 26, 2015 at 20:08
  • Protip: reiserfs for small files
    – dmourati
    Jun 28, 2015 at 19:10

2 Answers 2

2

When you have a problem like this, you have to:

  • obtain all requirements (latency, bandwidth, redundancy, reliability, security, required features...)
  • analyse the current systems. If they are none, create test environments. Understand how all components work. Understand the current and the expected load.
  • add system monitoring (with graphs) for both production and test systems. Monitor at least CPU usage, network usage and Disk I/O usage.
  • create test servers and load test them. Load test with synthetic benchmarks not only with micro-benchmarks.

Use stable versions of recent OS with latest stable kernel.

2
0

What would be a recommended file system setup in this scenario?

is there a file system particularly well suited for this case (e.g. ext4, xfs, btrfs, ...)?

I think ext4 is the best choice here in terms of safety. In terms of reading performance RaiserFS(/4) could be better:

https://unix.stackexchange.com/questions/28756/what-is-the-most-high-performance-linux-filesystem-for-storing-a-lot-of-small-fi

http://www.phoronix.com/scan.php?page=article&item=linux_39_fs&num=1

filesystem for millions of small files

what kind of RAID configuration (e.g. sw vs. hw, RAID level etc.) should we use?

You need to:

  • shorten latency / access time
  • reduce number of reads/writes (noatime,nodiratime,realtime,barrier=0)

Every RAID level, which has possibility to do parallel operations will be good choice, for example RAID 10 (Linux software raid) has a lot tuning options.

what about the file sharing setup (technology [e.g. NFS vs. competition], configuration, ...)?

I can't give you numbers here, file sharing is also quite wide term. I've had very good experience with FTP when dealing with lots of files (not FTPS - because negotiating TLS for every transfer takes time), better tunnel the traffic. It seems that the best division between flexibility of the protocol (what u can do) and protocol overhead (how much time it takes, space in the packet) when coping small files.

3
  • We'd basically need a distributed file system akin to NFS, but we're open other options if something else would be a better fit in our case.
    – marko
    Jun 26, 2015 at 20:13
  • Sorry, but it means nothing. What do you need? Block device sharing support, extended ACL's, xattr, compression? It sounds like you didn't analyze it yet. Jun 26, 2015 at 20:49
  • We need to share files across multiple systems. Currently, there's only one system that changes the filesystem at a given time and the filesystem contents need to be readable from two other systems. Compression or ACLs are not needed.
    – marko
    Jul 2, 2015 at 6:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .