I am administering a big LAMP server with some thousand users. About a week ago, things slowed down, and the only thing I see that the IO latency is increased dramatically. Users experience slow page loads, and I experience seconds of hanging when I want to save a file.
The operating system is CloudLinux, kernel 2.6.32. On top of that a wonderful combination of CageFS and cPanel. The hardware is an IBM X3630 M3, with 11 drives in hardware RAID 5 + a spare drive.
I did a lot of experiments. First, I ran
iotop -oaP to see what is doing a lot of IO bandwidth. All the processes that ended up in top positions are normal LAMP services. Those did not seem to do much more IO than they should - though I do not know the ideal or normal stress on the server. Sadly I can not access sysstat information from the days when the IO latency was normal, only the munin graphs. On the other hand, CageFs should limit all users' activity.
So I started to think, that the disks get a lot of IOPS, that they can not handle. The proprietary
megacli utility says no malfunction about the RAID array, no rebuild is in progress or anything unusual. Running
sar for hours I experienced IOPS over 5000, but the hangs are still there when the system is doing less than 1K IOPS, so I guess the disks are fine?
I've tried the audit framework and system tap, but both failed to be useful (the former hangs the whole system and I could not get much statistics, the latter did not even work at all).
What I am doing now is comparing my tiny laptop's speed to the server with several tests. That's how I found out, that while I can create 100K files with the following script on my laptop (with a small, laggy HDD) in 3-5 seconds, the server does it in more than 20-30 seconds.
#!/bin/bash i=1 while (( $i < $1 )); do echo $i echo "foobartest" > tmp/iotest.$i (( i++ )) done
This might be due to the server serving a 50-100 HTTP requests per second, but the strange thing is that if I observe the running numbers in the terminal sometimes it hangs for several seconds, before it can create the next file.
The thing I am currently doing is using
strace -T and parsing the output to see how long each syscall is hanging (as I can not use
What I found is open, write and dup2 are taking way more time than the others. All three are normal, given that I want to create a lot of files with content - so I do not really know where can I go forward?!
open 26,8320000000 write 11,5165000000 dup2 7,0665500000
NOTE: Upon request, I can upload outputs of commands like
Sorry for the poor english, it's 2 AM here, when noone really cares about the his/her website. Thank you in advance.
UPDATE: We changed the power supplies from double ~400W to double ~650W, and I do not experience the lag anymore. However, the latency is still high enough to be worried.
megacli showsummary a0 shows a problematic BBU:
Hardware Controller ProductName : ServeRAID M5015 SAS/SATA Controller(Bus 0, Dev 0) SAS Address : xxxx FW Package Version: 12.12.0-0047 Status : Need Attention BBU BBU Type : iBBU Status : Replace Battery pack