Odd variance in disk performance testing results

Question

I started out trying to do some simple performance tests. "Which of these disks is faster, and by how much" kind of things. I didn't need much precision, I just wanted a rough idea. It's turned into a weird foray into the internals of How Things Work. Kind of fun, actually, but a little odd.

Before I go further, I'll say that everything in this article is done on an Arch Linux system, running a 3.0 kernel, with a Western Digital WD30EZRX 3TB SATA disk drive removed from a MyBook Essential USB3 enclosure (those enclosures do evil things to your data) and installed in a Sans Digital single-disk USB3 enclosure, connected via a Buffalo USB3 PCIe 1-lane card. The drive is the only device on that USB card, and the system is otherwise idle during the time of the tests.

I wrote a performance testing program, a simple C program to open a file, write four GiB of data into it, fsync(), and exit. I ran the test 9 times (an arbitrary number, because I wanted single digits, but forgot to start at zero) on an empty whole-disk ext4 filesystem, and got 58-59 seconds per run, about 73 MB/sec. Not the fastest disk in the world, but the results were fairly consistent.

For reasons I no longer remember, not expecting any trouble, I came back and did the exact same test a few hours later, after unplugging the disk to test another one, plugging the original disk back in, and re-doing the mkfs. Much to my surprise, I got very different results. This time, I got results in the 51-52 second (83 MB/sec) kind of range, with 2 closer to 48 seconds.

Hours later, after doing some research, including asking a question here about how to find out which disk blocks are used by a file, I ran another test on the same disk without reinitializing. I just created new files in the same directory (the root of the filesystem) as the previous test. This time, I got results in the 44-45 second (96 MB/sec) kind of range.

Later on, I ran three more tests (refining my test procedure, actually), again just putting the files next to the existing ones, and got 44-second, 45-second, and 50-second performance. Those three groups of runs were all right next to each other. Keep in mind, I'm still not talking about individual runs. I ran the test program 9 times, all with 44-second results, another 9 times, all with 45-second results, and 9 more times, all with 50-second results.

These are some pretty big differences, and very oddly grouped.

I used debugfs to check which blocks the files are using, and there's nothing "funny" going on there. Each file seems to get the next available blocks, in sequence, starting from the beginning of the disk.

The only thing I can think of that might explain this would be if the disk circuitry were somehow allocating blocks from different parts of the disk and presenting them to the host as contiguous. For the results I'm seeing, it would have to be noticing the delay between my test runs and using that as an opportunity to switch to a different part of the disk for allocation.

Frankly, this sounds like a bit of a stretch to me, but I don't have a lot of other ideas.

Can anyone shed light on this? Why am I getting such locally-consistent yet globally-variable results?

Answer 1

That's a Caviar Green drive. I've heard of some inconsistent performance out of those thanks to the green circuitry. They have a variable RPM speed to save power, and that can absolutely affect transfer-rate. Enough for what you're seeing? I couldn't say, I haven't dealt with them personally. If consistent performance is what you're after, I'd go for their Blue or Black lines.

Answer 2

This is pretty typical for disks with a constant RPM (which almost all have because changing the rotation speed would slow seek times to a crawl). When the head is near the center of the drive, the linear speed is low and when the head is near the edge of the drive, the linear speed is higher. Most drives map the outer tracks to higher linear sector numbers and most file systems work their way up the linear sectors. So as you keep using a filesystem, it tends to get faster.

Answer 3

Months later, it seems worth following up on this still-open question of mine.

I had to shelve the project for a time, shortly after running into this roadblock, due to other priorities. Recently, I was able to get back to it.

The most frustrating thing was (and is) that I am now completely unable to reproduce the problem. I still have exactly the same hardware, and the same machines. However, all performance is reasonably consistent and as fast as I expect. For simplicity, my initial question did not include all the different setups (and hardware) I tried.

In particular, there was one machine which had the problem and which I simply shut down in the middle of testing this issue. Months later, it was still off. I turned it back on and resumed trying to characterize the inconsistent performance, only to find that the issue was gone.

Phase of moon? Number of the year? Sunspots? I have no idea. I have since re-kickstarted that machine, run numerous tests with two different USB interface cards and two different disks. No results are out of the ordinary. I'd have suspected my own sanity, except that I still had the cut-and-pasted logs from the original tests, along with the timestamped test output files. They were, without question, on that machine, on that disk, with odd results. Odd results that I am now unable to reproduce despite hours of trying.

I declare it to be not worth any more of my time, and I declare the question closed.

My apologies to those who come after me hoping for a definitive answer.