2

I have a cpanel LAMP dedicated server and I've been having load problems the last 2 days.

This is what my top looks like (sorted by M):

top - 14:26:04 up 1 day,  1:08,  2 users,  load average: 33.10, 36.63, 38.92
Tasks: 359 total,   1 running, 355 sleeping,   1 stopped,   2 zombie
Cpu(s):  4.2%us,  0.8%sy,  0.0%ni, 13.6%id, 81.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   1034896k total,   998084k used,    36812k free,     8716k buffers
Swap:  2040212k total,  1606552k used,   433660k free,    87388k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5088 mysql     15   0  336m 106m 3868 S  1.3 10.6 310:15.37 mysqld
15797 nobody    18   0  331m  65m 1988 S  3.0  6.4   0:04.64 httpd
15635 nobody    19   0  371m  63m 1840 S  0.0  6.3   0:00.88 httpd
15664 nobody    18   0  374m  63m 1832 S  1.3  6.2   0:00.43 httpd
15769 nobody    19   0  336m  59m 1700 S  0.0  5.9   0:00.29 httpd
15721 nobody    18   0  324m  59m 1732 S  1.0  5.9   0:00.29 httpd
15697 nobody    18   0  304m  59m 1692 S  0.0  5.8   0:00.46 httpd

iostat is:

Linux 2.6.18-164.15.1.el5 (hostname)   05/20/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          21.93    0.33    4.91   12.03    0.00   60.79

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              65.16      1069.51      1285.65   96981151  116580545
sda1              0.00         0.03         0.00       3023         31
sda2             18.59       444.95       159.37   40347535   14451402
sda3              7.15       129.47       113.45   11740498   10287608
sda4              0.00         0.00         0.00          6          0
sda5              1.25        12.88        44.49    1167658    4034632
sda6              3.92        11.25       525.79    1020250   47677744
sda7              5.56       108.71        96.29    9857739    8731832
sda8             28.70       362.20       346.25   32843994   31397296

Mysql seems to have many processes running (is constantly on "Too many connections"). None of my sites seems to have too much traffic and I see no specific queries taking more than others.

Could you please propose a way to debug / deal with this situation?

2
  • just running "iostat" once will give you average values since system startup. To get current values use iostat <interval> where <interval> is your time interval in seconds to average the data. Using the -x parameter to iostat further helps debugging as it provides you with other interesing data like queue length, service times and device utilization. To get hold of the process that is doing the most I/O use iotop
    – the-wabbit
    May 20, 2011 at 13:31
  • As you said you have been having this issue for the last 2 days, It might be some bug in some of your software? What have you changed on your server recently? Its clear from your top and iostat outputs that something is doing a lot IO activity, and as @synticon-dj mentioned you need to run iotop, but there are very low chances that your kernel would have support for it. May 20, 2011 at 13:44

5 Answers 5

3

I would guess your load issues stem from your excessive use of swap space due to running out of memory. As soon as your applications use up all the available RAM (1GB) they are going to start using the swap space (1.6 of 2GB used) which will increase your IO load (81.0%wa).

You almost never want your LAMP server to have to use swap space as, as you've noticed, it completely cripples the server's performance. In order to not use swap you have to limit the memory usage of your application:

  • Reduce the maximum number of Apache clients, typically with MaxClients. With only 1GB RAM you probably want to limit Apache to using 500MB or less which means a MaxClients of only 8 may be needed (500MB/60MB per process = 8). You can play with this number and if the server begins to swap reduce it and restart Apache.
  • Possibly reduce MySQL memory usage. Since you only have 1GB of RAM you may wish to limit MySQL usage. From your top output it seems fine at the moment but if it increases too high you may have to play with the configuration. The "right amount" of RAM to give to MySQL depends on your database and application. I might give a heavy db application 500MB but a very light one only 50MB.
  • Monitor the memory usage what ever other applications you have running. Having only 1GB on a LAMP server limits how much memory you can give to everything which, ultimately, will limit your serving capacity.
1
Cpu(s):  4.2%us,  0.8%sy,  0.0%ni, 13.6%id, 81.0%wa,  0.0%hi,  0.3%si,  0.0%st

See the very high value for wa? That's your problem. You're experiencing very high IO contention, and you have a lot of processes sitting around in the scheduler queue waiting for disk IO to happen.

I would recommend spending some time examining your disk situation. It's likely that you need to add additional spindles or faster disks.

1

IO wait is super high. It's not possible to determine from the top and iostat outputs what is killing disk. I would run iotop to see if you can find the IO culprit.

Also check "show full processlist" in MySQL to get an idea of what queries are running. It could be one or more queries bogging things down.

Lastly, enabled server status in Apache and check access logs to get a view into what Apache is trying to do. Maybe someone is scraping over and over?

The following may indicate if you're getting slammed by one host. It should display all current connections to port 80 ordered by count

netstat -anp |awk '/:80/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
0

"Too many connections" - means you need increase value of max_connections in mysql config. It may be result of many apache processes trying to connect to mysql server and stuck waiting when reach mysql max_connections limit. May be that application side does not close connection properly, or you if it's normal load you need increase limits. I think you need to monitor where mysql connections coming from, and how your application handle/close mysql connections.

0

Thank you all for the answers. They were all valid and helpful.

The actual culprit was actually cpanellogd - the process that cpanel runs to rotate and create stats out of access logs.

This was scheduled to run only at nights, but for some reason it was starting in the middle of the day when we are having the most load.

Not the answer you're looking for? Browse other questions tagged .