3

I am administering a big LAMP server with some thousand users. About a week ago, things slowed down, and the only thing I see that the IO latency is increased dramatically. Users experience slow page loads, and I experience seconds of hanging when I want to save a file.

The operating system is CloudLinux, kernel 2.6.32. On top of that a wonderful combination of CageFS and cPanel. The hardware is an IBM X3630 M3, with 11 drives in hardware RAID 5 + a spare drive.

I did a lot of experiments. First, I ran iotop -oaP to see what is doing a lot of IO bandwidth. All the processes that ended up in top positions are normal LAMP services. Those did not seem to do much more IO than they should - though I do not know the ideal or normal stress on the server. Sadly I can not access sysstat information from the days when the IO latency was normal, only the munin graphs. On the other hand, CageFs should limit all users' activity.

So I started to think, that the disks get a lot of IOPS, that they can not handle. The proprietary megacli utility says no malfunction about the RAID array, no rebuild is in progress or anything unusual. Running sar for hours I experienced IOPS over 5000, but the hangs are still there when the system is doing less than 1K IOPS, so I guess the disks are fine?

I've tried the audit framework and system tap, but both failed to be useful (the former hangs the whole system and I could not get much statistics, the latter did not even work at all).

What I am doing now is comparing my tiny laptop's speed to the server with several tests. That's how I found out, that while I can create 100K files with the following script on my laptop (with a small, laggy HDD) in 3-5 seconds, the server does it in more than 20-30 seconds.

#!/bin/bash

i=1
while (( $i < $1 )); do
    echo $i
    echo "foobartest" > tmp/iotest.$i
    (( i++ ))
done

This might be due to the server serving a 50-100 HTTP requests per second, but the strange thing is that if I observe the running numbers in the terminal sometimes it hangs for several seconds, before it can create the next file.

The thing I am currently doing is using strace -T and parsing the output to see how long each syscall is hanging (as I can not use stap).

What I found is open, write and dup2 are taking way more time than the others. All three are normal, given that I want to create a lot of files with content - so I do not really know where can I go forward?!

strace statistics:

open  26,8320000000
write 11,5165000000
dup2  7,0665500000

NOTE: Upon request, I can upload outputs of commands like sar etc. Sorry for the poor english, it's 2 AM here, when noone really cares about the his/her website. Thank you in advance.

UPDATE: We changed the power supplies from double ~400W to double ~650W, and I do not experience the lag anymore. However, the latency is still high enough to be worried.

Output of megacli showsummary a0 shows a problematic BBU:

Hardware
        Controller
                 ProductName       : ServeRAID M5015 SAS/SATA Controller(Bus 0, Dev 0)
                 SAS Address       : xxxx
                 FW Package Version: 12.12.0-0047
                 Status            : Need Attention
        BBU
                 BBU Type          : iBBU
                 Status            : Replace Battery pack
2
  • 1
    What kind of memory usage are you running into? It's common with MySQL to get really slow once you start using more database than you can cache. Are you hitting the pagefile?
    – Bert
    Dec 11, 2013 at 3:03
  • @Bert, the machine has plenty of RAM avialable. Only 20-30 GB is used from the available 64 GB. A lot of memory is allocated for caches and buffers, but alltogether they do not sum up to 64 GB. Swappiness is set to 60, still only 72 MB on the swap.
    – hgj
    Dec 23, 2013 at 10:33

1 Answer 1

2

the strange thing is that if I observe the running numbers in the terminal sometimes it hangs for several seconds, before it can create the next file.

This smells like you're filling up the write cache on your RAID controller. You do have a write cache, yes? (megacli showsummary a0)

Especially do check to see if your BBU is still optimal. In the default configuration, a failed/failing BBU is the same as no write cache.

Watch iostat to see if the hard drive busy% goes up to near 100% when things slow down.

More information such as underlying filesystem would also be helpful. Post graphs! Everything that you have! (well, most of it)

4
  • I could not get MegaCli to show me more cache information than: "Disk Cache Policy : Enabled" and "Is VD Cached: No" sar reports nearly 100% %util for /dev/sda, that is the RAID array. Here is a quick screenshot from the munin graph.
    – hgj
    Dec 23, 2013 at 10:38
  • 1
    No write cache means VERY slow performance, especially on a 11-disk RAID5. Given this suddenly happened a week ago, I'd wager that your write cache BBU gave up around then. Please post output of megacli showsummary a0.
    – MikeyB
    Dec 24, 2013 at 20:27
  • It was indeed a missing/broken BBU. It will be replaced and for the time being the leadership decided to force enable the cache. Please edit your post mentioning the BBU problem, and I will be happy to accept as solution. Cheers Mikey
    – hgj
    Feb 3, 2014 at 14:40
  • Done! Please also post the output so that it's indexed and available for people to search for. #forthechildren
    – MikeyB
    Feb 3, 2014 at 18:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .