Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm stumped and I hope someone else will recognize the symptoms of this problem.

Hardware: new Dell T110 II, dual-core Pentium G850 2.9 GHz, onboard SATA controller, one new 500 GB 7200 RPM cabled hard drive inside the box, other drives inside but not mounted yet. No RAID. Software: fresh CentOS 6.5 virtual machine under VMware ESXi 5.5.0 (build 1746018) + vSphere Client. 2.5 GB RAM allocated. The disk is how CentOS offered to set it up, namely as a volume inside an LVM Volume Group, except that I skipped having a separate /home and simply have / and /boot. CentOS is patched up, ESXi patched up, latest VMware tools installed in the VM. No users on the system, no services running, no files on the disk but the OS installation. I'm interacting with the VM via the VM virtual console in vSphere Client.

Before going further, I wanted to check that I configured things more or less reasonably. I ran the following command as root in a shell on the VM:

for i in 1 2 3 4 5 6 7 8 9 10; do
  dd if=/dev/zero of=/test.img bs=8k count=256k conv=fdatasync
done

I.e., just repeat the dd command 10 times, which results in printing the transfer rate each time. The results are disturbing. It starts off well:

262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 20.451 s, 105 MB/s
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 20.4202 s, 105 MB/s
...

but after 7-8 of these, it then prints

262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GG) copied, 82.9779 s, 25.9 MB/s
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 84.0396 s, 25.6 MB/s
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 103.42 s, 20.8 MB/s

If I wait a significant amount of time, say 30-45 minutes, and run it again, it again goes back to 105 MB/s, and after several rounds (sometimes a few, sometimes 10+), it drops to ~20-25 MB/s again.

Based on preliminary search for possible causes, in particular VMware KB 2011861, I changed the Linux i/o scheduler to be "noop" instead of the default. cat /sys/block/sda/queue/scheduler shows that it is in effect. However, I cannot see that it has made any difference in this behavior.

Plotting the disk latency in vSphere's interface, it shows periods of high disk latency hitting 1.2-1.5 seconds during the times that dd reports the low throughput. (And yes, things get pretty unresponsive while that's happening.)

What could be causing this?

I'm comfortable that it is not due to the disk failing, because I also had configured two other disks as an additional volume in the same system. At first I thought I did something wrong with that volume, but after commenting the volume out from /etc/fstab and rebooting, and trying the tests on / as shown above, it became clear that the problem is elsewhere. It is probably an ESXi configuration problem, but I'm not very experienced with ESXi. It's probably something stupid, but after trying to figure this out for many hours over multiple days, I can't find the problem, so I hope someone can point me in the right direction.

(P.S.: yes, I know this hardware combo won't win any speed awards as a server, and I have reasons for using this low-end hardware and running a single VM, but I think that's besides the point for this question [unless it's actually a hardware problem].)

ADDENDUM #1: Reading other answers such as this one made me try adding oflag=direct to dd. However, it makes no difference in the pattern of results: initially the numbers are higher for many rounds, then they drop to 20-25 MB/s. (The initial absolute numbers are in the 50 MB/s range.)

ADDENDUM #2: Adding sync ; echo 3 > /proc/sys/vm/drop_caches into the loop does not make a difference at all.

ADDENDUM #3: To take out further variables, I now run dd such that the file it creates is larger than the amount of RAM on the system. The new command is dd if=/dev/zero of=/test.img bs=16k count=256k conv=fdatasync oflag=direct. Initial throughput numbers with this version of the command are ~50 MB/s. They drop to 20-25 MB/s when things go south.

ADDENDUM #4: Here is the output of iostat -d -m -x 1 running in another terminal window while performance is "good" and then again when it's "bad". (While this is going on, I'm running dd if=/dev/zero of=/test.img bs=16k count=256k conv=fdatasync oflag=direct.) First, when things are "good", it shows this:

enter image description here

When things go "bad", iostat -d -m -x 1 shows this:

enter image description here

ADDENDUM #5: At the suggestion of @ewwhite, I tried using tuned with different profiles and also tried iozone. In this addendum, I report the results of experimenting with whether different tuned profiles had any effect on the dd behavior described above. I tried changing the profile to virtual-guest, latency-performance and throughput-performance, keeping everything else the same, rebooting after each change, and then each time running dd if=/dev/zero of=/test.img bs=16k count=256k conv=fdatasync oflag=direct. It did not affect the behavior: just as before, things start off fine and many repeated runs of dd show the same performance, but then at some point after 10-40 runs, performance drops by half. Next, I used iozone. Those results are more extensive, so I’m putting them in as addendum #6 below.

ADDENDUM #6: At the suggestion of @ewwhite, I installed and used iozone to test performance. I ran it under different tuned profiles, and used a very large maximum file size (4G) parameter to iozone. (The VM has 2.5 GB of RAM allocated, and the host has 4 GB total.) These test runs took quite some time. FWIW, the raw data files are available at the links below. In all cases, the command used to produce the files was iozone -g 4G -Rab filename.

The following is my summary.

In some cases I rebooted after a previous run, in other cases I didn’t, and simply ran iozone again after changing the profile with tuned. This did not seem to make an obvious difference to the overall results.

Different tuned profiles did not seem (to my admittedly inexpert eyes) to affect the broad behavior reported by iozone, though the profiles did affect certain details. First, unsurprisingly, some profiles changed the threshold at which performance dropped off for writing very large files: plotting the iozone results, you can see a sheer cliff at 0.5 GB for profile latency-performance but this drop manifests itself at 1 GB under profile enterprise-storage. Second, although all profiles exhibit weird variability for combinations of small file sizes and small record sizes, the precise pattern of variability differed between profiles. In other words, in the plots shown below, the craggy pattern in the left side exists for all profiles but the locations of the pits and their depths are different in the different profiles. (However, I did not repeat runs of the same profiles to see if the pattern of variability changes noticeably between runs of iozone under the same profile, so it is possible that what looks like differences between profiles is really just random variability.)

The following are surface plots of the different iozone tests for the tuned profile of latency-performance. The descriptions of the tests are copied from the documentation for iozone.

Read test: This test measures the performance of reading an existing file.

enter image description here

Write test: This test measures the performance of writing a new file.

enter image description here

Random read: This test measures the performance of reading a file with accesses being made to random locations within the file.

enter image description here

Random write: This test measures the performance of writing a file with accesses being made to random locations within the file.

enter image description here

Fread: This test measures the performance of reading a file using the library function fread(). This is a library routine that performs buffered & blocked read operations. The buffer is within the user’s address space. If an application were to read in very small size transfers then the buffered & blocked I/O functionality of fread() can enhance the performance of the application by reducing the number of actual operating system calls and increasing the size of the transfers when operating system calls are made.

enter image description here

Fwrite: This test measures the performance of writing a file using the library function fwrite(). This is a library routine that performs buffered write operations. The buffer is within the user’s address space. If an application were to write in very small size transfers then the buffered & blocked I/O functionality of fwrite() can enhance the performance of the application by reducing the number of actual operating system calls and increasing the size of the transfers when operating system calls are made. This test is writing a new file so again the overhead of the metadata is included in the measurement.

enter image description here

Finally, during the time that iozone was doing its thing, I also examined the performance graphs for the VM in vSphere 5’s client interface. I switched back and forth between the real-time plots of the virtual disk and the datastore. The available plotting parameters for the datastore were greater than for the virtual disk, and the datastore performance plots seemed to mirror what the disk and virtual disk plots were doing, so here I enclose only a snapshot of the datastore graph taken after iozone finished (under tuned profile latency-performance). The colors are a little bit hard to read, but what is perhaps most notable are the sharp vertical spikes in read latency (e.g., at 4:25, then again slightly after 4:30, and again between 4:50-4:55). Note: the plot is unreadable when embedded here, so I've also uploaded it to http://cl.ly/image/0w2m1z2T1z2b

vSphere real-time plot of VM

I must admit, I don’t know what to make of all this. I especially don’t understand the weird pothole profiles in the small record/small file size regions of the iozone plots.

share|improve this question
    
Assuming that you have been using high disk utilization in terms of IOPS for a long period of time, then you are likely experiencing disk saturation at some level (either at the os disk write queue level or more likely at the hard disk level) where you have more requests than your disk can handle at the moment. I am not experienced enough to tell you how to determine whether you are experiencing disk saturation but I think it might be a point of investigation for you. –  Jason Zhu May 29 at 4:31
    
@JasonZhu It's a good question. I've been assuming that because the system is doing nothing else, and I'm merely running the same command repeatedly, the level of disk usage would be roughly constant. Yet, it takes a long time before the behavior manifests itself despite this. I did run iostat and it showed ~90% utilization both before and after. But I'm not an expert in judging these things – maybe saturation is happening somewhere. I'm updating my question to show iostat output in case it's useful. –  mhucka May 29 at 14:25
    
Is there any chance this is due to ESX disk caching? Is the disk you're testing optimized for performance or safety? Also, if it's optimized for performance, can you look at the ESX cache utilization while you're testing? This write IO change in throughput looks like writes hitting a cache, and then slowing to the destage speed when it fills up. –  Basil Jun 3 at 16:52
    
Could that have anything to do with you RAID controller cache? High IOPS/throughput until it's full and then you fall back to raw HDD performance? Just guessing... –  Mario Lenz Jun 3 at 19:56
    
@MarioLenz As I mentioned in the description, this has no RAID. –  mhucka Jun 5 at 22:46

2 Answers 2

Can you give the exact ESXi build number? Please try tests again with a purpose-built disk performance analysis tool like fio or iozone to get a real baseline. Using dd is not really productive for this.

In general, the default I/O scheduler in EL6 isn't that great. You should be looking at shifting to the deadline or noop I/O elevators, or better yet, installing the tuned framework.

Try: yum install tuned tuned-utils and tuned-adm profile virtual-guest, then test again.

share|improve this answer
    
Argh, I neglected to mention that I did actually change the scheduler to noop. I'll edit my question to say that. As for build number, it's 1746018. (I did include that in the 2nd paragraph, but I see something happened and it was truncated -- must have been an editing accident. I'll fix that too.) I'll look into the tuned framework and the other tools. Thanks for your suggestions. –  mhucka May 29 at 16:36
    
I tried tuned, using profile virtual-guest and keeping everything else the same (proper experimental technique -- avoid changing more than one variable). It did not affect the behavior: just as before, things start off fine, but after many repeated runs (10-30) of dd if=/dev/zero of=/test.img bs=16k count=256k conv=fdatasync oflag=direct, performance drops by half. I also tried profile latency-performance -- same result. I'm currently trying throughput-performance. –  mhucka May 29 at 20:58
    
Can you try something that doesn't involve dd runs? Perhaps the fio or iozone mentioned earlier? –  ewwhite May 29 at 21:00
    
I will. But first, it made sense to repeat the same test, to see if the behavior changed. –  mhucka May 29 at 21:05

Try to find out where in your storage stack the high latencies are caused:

enter image description here

source: Troubleshooting Storage Performance in vSphere – Part 1 – The Basics

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.