Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've been trying to troubleshoot a problem with I/O on my disks. The setup is as follows:

  • OS: CentOS 5.6
  • Disk layout:
    • Disks (/dev/sda, /dev/sdb)
    • Partition (/dev/sda1, /dev/sdb1)
    • MD Array (RAID-1) (/dev/md0)
    • LVM Stack (/dev/VolGrp00/RootLV)

Initially, I noticed that when performing heavy I/O (i.e. mkfs), the system would slow to a crawl, to the point that I couldn't move the mouse pointer in my X session. I started logging some metrics and saw that the load average would slowly climb, up to above 5.0 on my dual-core server. At the same time, my memory picture goes from nearly 2GB free to about 10MB free and nearly 2GB buffer. Based on that, I suspect that some kind of caching is to blame, but I'm not familiar enough with the nuts and bolts of LVM, MD and the Linux I/O subsystem to know where to start looking.

One oddity I found: it doesn't seem to matter if I strip off the LVM layer and write directly to the array, and even removing the array doesn't help much (though writing directly to the partition seems to cause shorter bursts of delays than writing to the array).

Most of my testing has been done with the command mkfs.ext4 -i 4096 -m 0 <device>, though I have tested this behavior with dd if=/dev/urandom bs=4K of=<device> and received similar results so I'm fairly sure it's not the fault of mkfs. Also, I've tried this on another system (from another hardware vendor, but still CentOS 5.6) and again see similar results.

I'm okay with any solutions that cause my I/O operations taking a little longer to complete, though answers like "use the direct I/O flag" are unacceptable since they cause mkfs to go from 10 minutes to 16 hours (been there, tried that). I'm searching for tuning parameters and am also looking into changing I/O schedulers, but I figured it might be helpful to ask the community for some guidance in the right direction.

EDIT :

As it turns out, the problem is more related to memory pressure and the virtual memory manager causing I/O requests to block. Here's my current understanding of the issue: As mkfs or dd run, they generate more I/O than the disks can keep up with, so the buffer begins to fill up. As the vm.dirty_ratio is reached, I/O requests from all processes begin to block until the cache clears out some space (source). At the same time, low memory conditions trigger the kernel to start swapping processes out of physical memory and onto disk... this generates even more I/O and those I/O requests can block while waiting for the cache to clear.

I have tried tuning vm.dirty_ratio and other related parameters, but they only change when the system starts to slow down (lower ratio = faster lock-up). I've also tried swapping out I/O schedulers and tweaking their parameters to try to clear the cache faster but have had no success. As a last resort, I tried running the mkfs with ionice -c3, but since the disks are mostly idle at the time I'm running the mkfs, the same problem manifests itself. I think the slowdowns could be avoided if there were a way to throttle the I/O request rate of a specific process, but I'm not aware of anything that will do that.

I'm definitely still open to suggestions as far as what to try -- whoever can push me in the right direction gets the green checkmark.

ANOTHER EDIT :

I've stumbled across control groups, but they are unfortunately only available starting in RHEL6. Cgroups could be employed to start the mkfs in a group with throttled block I/O, but since these systems must remain 5.6 for the time being I will have to either keep looking for another solution or deal with the slowness issues until upgrade time.

share|improve this question
    
You only have one disk in the md array? Or, what happened to the second disk? –  Michael Hampton Jul 13 '12 at 23:27
    
So what is your hardware? Harddrive, controller, cable, mainboard, bios settings, and the kernel dmesg? –  Andrew Smith Jul 14 '12 at 1:28
    
@MichaelHampton: No, there are two disks (/dev/sda1 and /dev/sdb1). The above is supposed to be a visualization of the full stack. I'll edit the question to clarify this. –  Justin ᚅᚔᚈᚄᚒᚔ Jul 15 '12 at 16:02
    
@AndrewSmith: I've dried on different hardware, both an IBM SystemX 3250, Dell PowerEdge 2950 and Dell PowerEdge R410, with both SAS and SATA drives of differing speeds. The IBM and the 2950 have non-battery-backed LSI SAS controllers, and the R410 has a PERC6. NCQ is off in all cases (we have to turn this off due to an unrelated issue). I honestly don't believe this to be a hardware issue. –  Justin ᚅᚔᚈᚄᚒᚔ Jul 15 '12 at 16:07
    
While the only answer is a practical one (tune your Linux Kernel's IO Scheduler) I see this problem more on certain motherboards and chipsets, than on others. I see similar issues on a PC with a given motherboard and hard drive, that don't go away whether it's Linux or windows running. Anyways, you didn't post your motherboard or chipset, or model of drive. I am hardly surprised that software RAID isn't a good way to achieve maximum I/O bandwidth and user-responsiveness to the GUI simultaneously. –  Warren P Oct 19 '12 at 2:26

1 Answer 1

From the scanty details you've given us, it sounds to me that you need to tune your I/O scheduler. That can have a significant impact on the lockup effect you're experiencing. I believe CentOS 5.6 uses CFQ. You may get less locking with deadline.

share|improve this answer
    
My initial results with switching to deadline are curious indeed. I saw mostly the same behavior up to a point: load average went to 9, and the array and layers above showed vastly more tps than the disks themselves. But, after a few minutes, the load average dropped back down to around 2.1 and the system was responsive again, buffer memory was still around 1.5 GB and dropping slowly. I'm going to test this multiple times but it seems that the mkfs is filling up the buffer and then bogging the system down with I/O wait until the writes to disk catch up. –  Justin ᚅᚔᚈᚄᚒᚔ Jul 16 '12 at 14:27
    
get an SSD , see if improves your 'problems' ;-) –  The Unix Janitor Jul 20 '12 at 20:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.