6

We're building a product that is likely to generate very large XFS volumes, and I'm trying to discover the scaling bottlenecks we're likely to run into given the architecture.

As we manipulate files they get placed into directories on the XFS volumes. Due to the number of files we handle, the file-count is definitely in the tens of millions and will likely get into the hundreds of millions before too long after release. We know this because our current product behaves this way, so it is reasonable to expect our next one to do similarly.

Therefore, correct early engineering is in order.

This week the files are based on the following rough layout:

$ProjectID/$SubProjectID/[md5sum chunked into groups of 4]/file

Which gives directories that look kind of like:

0123456/001/0e15/a644/8972/19ac/b4b5/97f6/51d6/9a4d/file

The reason for chunking the md5sum is to avoid the "big pile of files/directories in one directory" problem. Due to the md5sum chunking it means that 1 file causes 8 directories to be created. This has pretty clear inode impacts, but I'm unclear on what those impacts will be for XFS once we get up to scale.

What are the impacts?

This is with kernel 2.6.32 by the way, CentOS 6.2 at the moment (this can change if needed).

In testing I've created the xfs volume with defaults, and am not using any mount-options. This is to smoke out problems early. noatime is something of a no-brainer as we don't need it. Overall XFS tuning is another problem I need to tackle, but right now I'm concerned about the metadata multiplier effect we've engineered in right now.


I already know what a better solution will be, I just don't know if I have a case to push for the change.

Since md5sums are significantly unique in the first digits, and individual sub-projects rarely exceed 5 million files, it seems to me that we only need the first two chunks. Which would yield layouts like:

0123456/001/0e15/a644/897219acb4b597f651d69a4d/file

A completely full first and second levels would have 216 first level directories and 216 second level directories in each first level directory, for a total of 232 directories on the volume.

The hypothetical 5 million file sub-project would therefore have 216 first level directories, roughly 76 (+/- 2) second tier directories in each, and one or two third tier directories in each second tier directory.

This layout is a lot more metadata efficient. I just don't know if is worth the effort to change how things are going right now.

5
  • 1
    Interesting question. I'd like to see what your mkfs.xfs formatting and mount options will be. Also, what types of operations will be performed on this dataset? I have 10+ million files in some of my production XFS systems. The operations are a mix of heavy read/write on large files and heavy read on millions of small files. Will you have rename/mv/delete type of activity on the filesystem?
    – ewwhite
    May 21, 2012 at 15:14
  • @ewwhite The original files will be read a few times, and processing products will be written along side them. There will be no renames, and deletes will be mass-deletes when they happen (prune an entire project tree). I/O is highly random, but each file will be handled sequentially. By far the biggest metadata op will be mkdir and create.
    – sysadmin1138
    May 21, 2012 at 15:34
  • Seeing as how it doesn't take a very large sub-project to get to 65535 directories in Tier 1, I suspect I may want to increase my inode block-size (-i size=X) up from the default of 256.
    – sysadmin1138
    May 21, 2012 at 15:37
  • And your agcount?
    – ewwhite
    May 21, 2012 at 15:50
  • @ewwhite 4 at the moment in test, but the production system should be larger. How much larger I haven't figured out yet. The storage subsystem is a large multidisk array (test is ~120 spindles shared widely in our environment, production will be a smaller spindle-count, but dedicated). A dedicated journal device is possible here.
    – sysadmin1138
    May 21, 2012 at 15:53

1 Answer 1

3

No major recommendations other than that XFS should scale to this. I started using the filesystem in 2003 because I needed to work around an application that could easily have 800,000 files in a single directory. ext2 and ext3 would routinely fall over in operations within those filesystems.

A lot of this depends on your application and how it accesses files (directory traversal, etc.).

If this is all on one server, I would look at external SSD journals based on your expectation of a high number of metadata operations. But you know that part. I would still push for the restructuring using the second md5 example. I mean, this is a good time to refactor, right?

3
  • I realized after posting that when it comes time to remove a whole project, deleting all those inodes will make the delete-op slow. XFS is fast for deletes, but rmdiring 8 directories takes longer than 3 no matter how you cut it.
    – sysadmin1138
    May 21, 2012 at 15:55
  • I'm curious... will these be fixed partitions or LVM?
    – ewwhite
    May 21, 2012 at 16:23
  • These will be on LVM. As we add storage, it seems the better option for handling expandability. It should be end-to-end 64-bit so > 2TB LUNs are doable.
    – sysadmin1138
    May 21, 2012 at 16:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .