We are running LAMP+memcached on an Ubuntu 9.10 x64 xlarge Amazon EC2 instance. This server handles a few hundred requests per second, of which about 60% are static and the remainder all interact with mysql and/or memcached in some way. This server has been suffering from two performance issues that are possibly related and have proven difficult to diagnose. All statistics below have been gathered with CloudWatch, munin or vmstat/iostat/top unless otherwise specified.
The first problem is recurring regular spikes of high iowait every few minutes during which most apache processes simultaneously iowait for about 10-30 seconds before all un-hanging. There is no increased disk or network load during this time, disk queue stays low, no swapping going on, etc.
More seriously, during peak times the server sometimes suddenly experiences a drastic loss of performance, dropping serviced requests to ~1/3 of what it was a moment before. Once started, this performance dip can last anywhere from 2 to 8 hours before suddenly springing back up to full performance again. When this happens, it's as if the system just stops doing anything. CPU utilization, disk load and network load (as reported by CloudWatch) all simultaneously drop proportionally, yet there is no disk contention. Disk queue and throughput both drop and are well below maximum at all times, especially during these dips. EDIT: This problem has been resolved. Apache was running out of worker processes and for some reason decided this was a good reason to crash performance entirely, even for those processes that were working fine.
The exception is network reads, which remain as high as before, indicating that the server is still being accessed in as high volume as before. If we attempt to contact the server ourselves when this happens, the server is extremely slow and often simply drops the connection before the request can be serviced. It should be noted that neither memory usage or CPU utilization is especially high at any time, whether or not performance is currently tanking: CPU% rarely goes above 10%, the disk is not full or congested. We have not been able to gather data on swap performance during these dips yet, but are attempting to do so.
As it is, we are running low on ideas for what could be causing these mysterious issues and are increasingly worried that this may be a problem (or misfeature) of EC2 itself. The fact that the massive dips always seem to occur when our traffic is peaking (though, again, this does not mean the server is close to maxing out its available resources) cannot simply be coincidence.
All MySQL databases and logs are hosted on an EBS volume and all static content is hosted on a separate, different EBS volume. Apache services 160-240 requests per second and MySQL 180-200 queries per second, with ~0% slow queries and a ~90% hit rate from memcached. Load average tends to hover around 3. Apache access logging is disabled to minimize disk access.