I started out trying to do some simple performance tests. "Which of these disks is faster, and by how much" kind of things. I didn't need much precision, I just wanted a rough idea. It's turned into a weird foray into the internals of How Things Work. Kind of fun, actually, but a little odd.
Before I go further, I'll say that everything in this article is done on an Arch Linux system, running a 3.0 kernel, with a Western Digital WD30EZRX 3TB SATA disk drive removed from a MyBook Essential USB3 enclosure (those enclosures do evil things to your data) and installed in a Sans Digital single-disk USB3 enclosure, connected via a Buffalo USB3 PCIe 1-lane card. The drive is the only device on that USB card, and the system is otherwise idle during the time of the tests.
I wrote a performance testing program, a simple C program to open a file, write four GiB of data into it, fsync(), and exit. I ran the test 9 times (an arbitrary number, because I wanted single digits, but forgot to start at zero) on an empty whole-disk ext4 filesystem, and got 58-59 seconds per run, about 73 MB/sec. Not the fastest disk in the world, but the results were fairly consistent.
For reasons I no longer remember, not expecting any trouble, I came back and did the exact same test a few hours later, after unplugging the disk to test another one, plugging the original disk back in, and re-doing the mkfs. Much to my surprise, I got very different results. This time, I got results in the 51-52 second (83 MB/sec) kind of range, with 2 closer to 48 seconds.
Hours later, after doing some research, including asking a question here about how to find out which disk blocks are used by a file, I ran another test on the same disk without reinitializing. I just created new files in the same directory (the root of the filesystem) as the previous test. This time, I got results in the 44-45 second (96 MB/sec) kind of range.
Later on, I ran three more tests (refining my test procedure, actually), again just putting the files next to the existing ones, and got 44-second, 45-second, and 50-second performance. Those three groups of runs were all right next to each other. Keep in mind, I'm still not talking about individual runs. I ran the test program 9 times, all with 44-second results, another 9 times, all with 45-second results, and 9 more times, all with 50-second results.
These are some pretty big differences, and very oddly grouped.
I used debugfs to check which blocks the files are using, and there's nothing "funny" going on there. Each file seems to get the next available blocks, in sequence, starting from the beginning of the disk.
The only thing I can think of that might explain this would be if the disk circuitry were somehow allocating blocks from different parts of the disk and presenting them to the host as contiguous. For the results I'm seeing, it would have to be noticing the delay between my test runs and using that as an opportunity to switch to a different part of the disk for allocation.
Frankly, this sounds like a bit of a stretch to me, but I don't have a lot of other ideas.
Can anyone shed light on this? Why am I getting such locally-consistent yet globally-variable results?