Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Fedora box with some severe IO limitations which I have no idea how to debug.

The server has a Areca Technology Corp. ARC-1130 12-Port PCI-X to SATA RAID Controller with 12 7200 RPM 1.5 TB disks and a Marvell Technology Group Ltd. 88E8050 PCI-E ASF Gigabit Ethernet Controller.

uname -a output: 2.6.32.11-99.fc12.x86_64 #1 SMP Mon Apr 5 19:59:38 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

The server is a file server running Nginx with the stub status module enabled, so I can see the current amount of connections. The problem present itself when I have a high number of simultaneous connections in a writing state. Usually around 350, at this very moment it's at 590 and the server is almost unusable and stuck at 230mbit/s.

If I run stop and hit 1 to see CPU core usages I have all 4 cores with around 99% io wait, if I run iotop the nginx workers are the only processes producing any read load, currently at around 25MB/s. I have each of the workers bound to their own core.

Initially I figured it was just the disks being bugged. But I've run fscheck and smartmontools checks and found no errors. I also ran an iozone test which you can see the result of here: http://www.pastie.org/951667.txt?key=fimcvljulnuqy2dcdxa

Additionally, when the amount of connections are low I have no problem getting a good speed. If I wget over the local network it easily hits 60MB/sec.

Right now I just tried putting a file in /dev/shm, then I symlinked a file from the public dir to it and used wget over the local network and only got 50KB/s.

Also, if I try to cp /dev/shm/test /root/test it quickly copies around 740MB and then slows down HEAVILY. Again with iotop reporting 99% iowait.

I'm not really sure how to go about figuring out what the problems are. It could be a natural disk limitation but then the file from /dev/shm ought to transfer so it seems there's a network limit, but that's fine when there's not many connections. Perhaps it's a TCP stack problem but I really have no idea how to check that.

Any suggestions on how to proceed with debugging would be very welcome. If additional information is required then let me know and I'll try to get it.

Thanks.

share|improve this question
    
What level and style of IO writes do each of those 350 connections create? How are the disks configured (RAID 5\10?). Those disks can sustain a bit more than 1000 IOPs at best - that could easily not be enough for the load the connections are creating. –  Helvick May 8 '10 at 19:20
    
Disks are in RAID 10. The connections are end home users so could be anything from 5kb to 2MB/s. The most throughput I've managed to get out of a connection while the server is under load has been 50KB/s whether remotely or over local network. The thing is this happens even when reading a file from memory. I actually just tried using rsync over ssh and the memory file transfers at 50MB/s to a normal disk on another server. So it should be something network related which manifests itself in iowait, right? –  Martin Fjordvald May 8 '10 at 19:45
add comment

1 Answer

up vote 1 down vote accepted

iotop is nice for seeing which processes are creating io, but I'd use sar for some more specific numbers; sar -d 10 6, for example, will give you 10 second samples across a minute period which give you much more detail on your disk performance and whether you actually have bottlenecks there (bear in mind that quite small await/svctime can have significant impacts on performance - I've seen as little as 20ms svctimes render a database server ususable, since that's 20ms per IOP the DB was trying to do).

Beyond that setting up sar's sa1 (in /etc/cron.d/sysstat) to collect more frequently than the every ten minute default and getting a full dump of the stats in gathers during busy periods (sar -A -s 09:00:00 -e 10:00:00) will give you detail on the network performance as well, and make it easy to correlate CPU, disk, network, memory behaviour to look for dodgy numbers.

(And yes, network can show up as iowait)

share|improve this answer
    
Not entirely sure how to read this, but this is the output: pastie.org/951832.txt –  Martin Fjordvald May 8 '10 at 21:25
    
See those await times? They don't look so good to me. That means each request you try to pull from or send to the disk is averaging 80ms to be satisfied. That's most likely what's killing your performance. avgqu-sz is the size of the queue for that disk - you've got, on average, nearly 16 IO operations queued up for the disc. dev253-0 is a logical volume, I'm guessing residing on /dev/sda (dev8-0). It's the bottlenecked disk. –  Rodger May 8 '10 at 22:49
    
And this is likely to be due to disks being satuated? Is there any other possible factors here? I'm currently only using 1.2TB of the 7.9TB available, so perhaps restructure the RAID 10 setup to get more read performance? –  Martin Fjordvald May 8 '10 at 23:26
    
The disk devices in question are certainly saturated but I don't think they should be. It looks out of whack - there's only about 200 IOPS reported but if you have 12 7.2k disks in a working RAID 10 pack it should be capable of handling 1000 read IOPS before degrading like this. Are you sure the RAID pack is healthy? –  Helvick May 9 '10 at 0:16
1  
At a minimum you should be able to launch the ARC\McBIOS RAID Manager when the machine boots or via a browser if you have the driver proxy installed and then check its event log and that the status of the RAID set is not degraded. The ARC-1130 manual ( areca.com.tw/support/main.htm ) points at a lot of things that could be tweaked (Read ahead cache, NCQ tagging, SATA mode for starters). –  Helvick May 9 '10 at 1:27
show 4 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.