Server I/O climbs, bandwidth peaks, server locks, where to start?

Question

I'm using Linode to host a Wordpress blog with an eCommerce plugin named 'Shopp'.

I moved the site just a few days ago from a previous shared hosting account (with less resources than the Linode VPS).

The problem did not occur on the previous host.

Every few hours (around 10-11 hours, varies) I can see a peak in bandwidth ('normal' traffic is around 400k-600k/s) of 1.1mb/s. At the same time I can see that I/O and SWAP also peaks to between 10k-20k which causes the server to basically lock down. It locks for around 30 to 40 minutes or until I reboot it.

I've turned on the mysql slow query log to see if I can get any clues there, but unfortunately all the queries it logged seemed normal (things like 'select column, from table1 where id=1', another example that caught my eyes on the slow query log is 'set names utf8').

How can I investigate what is causing this issue? Is there a way to know for sure if the problem originates from the db, or maybe one of the php files on the server?

What's the best way to continue from here?

P.S: Running Ubuntu 10.04, 32bit, latest mysql/php/apache2 and using Linode 1024

Answer 1

Figuring out an issue like this is easiest when the server is having one of the bad moments. It helps if can pin it on CPU, memory, or disk.

• Install and run htop. Check the load average numbers at top right. Sort (press F6) by CPU% and then MEM% and see if any outliers.
• If you have an outlier and it is an apache/httpd process run "links http://localhost/server-status" and see if can figure out which page causes the problem
• If that doesn't help, "strace -p PID" the stray process to see if you can figure out which resources it is trying to access and what it might be doing.
• Run vmstat 1, watch the "wa" column near the far right. That is IO wait, if it is above 20 or 30 IO is most likely the issue.

It does sound like memory is the issue though. Perhaps the mysql configuration on the new host is too aggressive? Make sure your key_buffer and innodb_buffer_pool_size combo isn't too much of your RAM, especially since it isn't a dedicated DB machine and apache needs a fair amount of memory to continue providing httpd processes. Mysql will over time use every bit of memory you allow it so could be it takes 10-11 hours for those buffers to fill up.

I'd check your keepalive settings in apache as well, those can have a huge impact on memory usage. I usually set that pretty low or turn it off entirely.

Answer 2

There's a lot of QA resources at your disposal on this site, first you need to find out what your actual limiter is, you mention SWAP so I'm guessing your having a memory issues.

To get better help, you need to provide some diagnostic info for people who might want to help, e.g.: output from top for starters.

It might sound wrong, but you can improve performance and behavior by tuning your Apache/MySQL installs to work better within the hardware constraints.