I am administering a big LAMP server with some thousand users. About a week ago, things slowed down, and the only thing I see that the IO latency is increased dramatically. Users experience slow page loads, and I experience seconds of hanging when I want to save a file.
The operating system is CloudLinux, kernel 2.6.32. On top of that a wonderful combination of CageFS and cPanel. The hardware is an IBM X3630 M3, with 11 drives in hardware RAID 5 + a spare drive.
I did a lot of experiments. First, I ran iotop -oaP
to see what is doing a lot of IO bandwidth. All the processes that ended up in top positions are normal LAMP services. Those did not seem to do much more IO than they should - though I do not know the ideal or normal stress on the server. Sadly I can not access sysstat information from the days when the IO latency was normal, only the munin graphs. On the other hand, CageFs should limit all users' activity.
So I started to think, that the disks get a lot of IOPS, that they can not handle. The proprietary megacli
utility says no malfunction about the RAID array, no rebuild is in progress or anything unusual. Running sar
for hours I experienced IOPS over 5000, but the hangs are still there when the system is doing less than 1K IOPS, so I guess the disks are fine?
I've tried the audit framework and system tap, but both failed to be useful (the former hangs the whole system and I could not get much statistics, the latter did not even work at all).
What I am doing now is comparing my tiny laptop's speed to the server with several tests. That's how I found out, that while I can create 100K files with the following script on my laptop (with a small, laggy HDD) in 3-5 seconds, the server does it in more than 20-30 seconds.
#!/bin/bash
i=1
while (( $i < $1 )); do
echo $i
echo "foobartest" > tmp/iotest.$i
(( i++ ))
done
This might be due to the server serving a 50-100 HTTP requests per second, but the strange thing is that if I observe the running numbers in the terminal sometimes it hangs for several seconds, before it can create the next file.
The thing I am currently doing is using strace -T
and parsing the output to see how long each syscall is hanging (as I can not use stap
).
What I found is open, write and dup2 are taking way more time than the others. All three are normal, given that I want to create a lot of files with content - so I do not really know where can I go forward?!
strace statistics:
open 26,8320000000
write 11,5165000000
dup2 7,0665500000
NOTE: Upon request, I can upload outputs of commands like sar
etc.
Sorry for the poor english, it's 2 AM here, when noone really cares about the his/her website. Thank you in advance.
UPDATE: We changed the power supplies from double ~400W to double ~650W, and I do not experience the lag anymore. However, the latency is still high enough to be worried.
Output of megacli showsummary a0
shows a problematic BBU:
Hardware
Controller
ProductName : ServeRAID M5015 SAS/SATA Controller(Bus 0, Dev 0)
SAS Address : xxxx
FW Package Version: 12.12.0-0047
Status : Need Attention
BBU
BBU Type : iBBU
Status : Replace Battery pack