8x slower cat for the same files on the same disk but different directory

Question

I'm having an issue where I have 8x slower access to a set of files as compared to the same files in another directory on a linux machine.

The filesystem is a 36TB RAID-5 filesystem exported from a Dell PERC H810, and it's formatted with ext4. The machine has 256GB RAM, and I'm using OpenSuSE 12.3 with kernel 3.7.10-1.45-desktop.

The problem is seen with something simple like "time cat slowdir/* > /dev/null", but "time cat fastdir/* > /dev/null" is about 8x faster. I'm clearing the IO cache between tests (echo 3 > /proc/sys/vm/drop_caches), so that shouldn't be affecting my results.

Both slowdir and fastdir are in the same filesystem and same parent directory.

Here are some more oddities about the problem. If I do the following, the problem persists in the new directory, alsoslowdir:

• cd /parentdir
• cp -r slowdir alsoslowdir
• echo 3 > /proc/sys/vm/drop_caches
• time cat alsoslowdir/* > /dev/null (BAD: takes 8 mins)

But, if I make a new directory, alsofastdir, and copy all of the files into it, then it's 8x faster with this method:

• cd /parentdir
• mkdir alsofastdir
• cp slowdir/* alsofastdir/
• echo 3 > /proc/sys/vm/drop_caches
• time cat alsofastdir/* > /dev/null (GOOD: takes 1 min)

All of the files are between 7 and 15MB in each of the directories and there are a few thousand files, with a total of 58GB in the dir.

I've checked the /usr/sbin/filefrag statistics for all of the files in the fast and slow directories and they are all either 1 or 2 extents, with about the same count of 1 & 2 extents between them.

What am I missing?

Answer 1

In your "fast" test, you showed that you copied recursively: cp -r slowdir alsoslowdir

In your "slow" test, you didn't copy with a recursive flag, but rather with a wildcard: cp slowdir/* alsofastdir/

Do you have any sub-directories in slowdir? Not 100% sure whether wildcard also includes sub-directories, but I'm fairly certain it does not, and only expands out to all matching "objects" in the directory, meaning that sub-directories would be left empty.

You may want to retry your test, and ensure with absolute certainty that the data contained by each directory is identical.

If that gets you nowhere... Perhaps just add "fast" to all of your directory names? (j/k) Look into finding a good performance testing tool though - cat really isn't a good method to measure IMO. Find a tool that allows for adjusting threads, I/O size, read/write mix, etc with tests being run on a specific file (sorry, no specific tool names come to mind at the moment).

By the way - what even led you to test performance on individual directories like you're doing? I'm sure there was some kind of odd behavior you encountered to start this off...

Answer 2

Granted that directories structure is absolutely the same (and you should really check that), your results can be skewed by caching (both at controller and OS level) and/or much different physical disk layout for the two directories.

Try that:

1. make sure that you have iostat binary (it's typically part of sysstat package)
2. invalidate PERC cache by issuing
   dd if=/dev/zero of=bigfile bs=4k count=1M oflag=direct; sync
3. invalidate OS cache by issuing
   sync; echo 3 > /proc/sys/vm/drop_caches
4. gather disk statistics issuing
   iostat -x -k 5 > stat.txt & cat dir/* > /dev/null; killall iostat
5. repeat all these steps for the other directory, upload you disk statistics (for both directories) and let me see them