Apache slows randomly

Question

I got a new dedicated server from singlehop with a 1Gps connection. For about a week the thing was a beast. Everything ran super fast. it is now dirt slow.... My $30/month godaddy vps is faster. I've tried all sorts of things (upgraded apache/php/reinstalled os/etc...) and nothing. Traceroute and pings don't reveal any problems either. I've also tried adjusting apache settings such as keepalive/maxclients/maxservers/etc.. with absolutely no change. It honestly feels like I'm on a pos hostv vps. A big $250/month dedi shouldn't act that way and I'm ticked to say the least (sorry for venting).

I wrote test scripts to load up my server before I decide to move my site over. I don't wanna move it over just to get burned again by another crappy hosting company. My test scripts should complete in around 8-11 minutes tops. Each consists of 2500 xml http requests and I run 4 of them simultaneously. Now they take 20+ minutes to complete (2-5x longer...) for no apparent reason (that I can see). There are no system resource hogs. None of my scripts use more than 0.5% cpu/0.7% mem each. There's no disk trashing or anything. The scripts don't fail, they just run slow. Sometimes they go faster at random.. but mostly just slow now.

I don't know if this means anything, but I ran a poor dumb man's speed test a few minutes ago. I opened up FileZilla FTP Client and dropped in 1008 25kb xml files. It took ~1 minute & 45 seconds to complete the transfer. I noticed my server was using pure-ftp for the process. This was done from my home computer to the server. I don't know if pure-ftp uses apache or not, but I'm leaning towards not.

ANY help or advice would be greatly appreciated!! I know server tuning can be difficult, but considering I just drag and drop the same test scripts onto a godaddy vps and they run fine, this is non-sense to me..

Thanks for any and all advice!!

Answer 1

I would try running a series of tests using ab (ApacheBench) on both servers. Make sure you run each on the relevant server, i.e., don't run both tests from the same server. For basic test parameters you can use something like:

ab -kc 10 -t 30 http://yoursite.com/page

which uses 10 concurrent connections, keep alive and performs as many requests as possible in 30 seconds. Afterwards some stats will be output with the important number being "Requests per seconds".

Run tests for:

• Static Pages: Images, or raw HTML files. You should get a very high request rate on both servers.
• Dynamic Pages: PHP or whatever type of dynamic content you are using. Use a variety of pages, some simple some complex, to get a range of request rates.

What to do next depends on the above results. Hopefully it will tell you which types of pages run slow on which server which will give you a starting point for further investigation (or another ServerFault question).

Other things I would look at:

• Compare the CPU/RAM of the two servers to ensure your dedicated server is actually more powerful. If unsure look into performing some basic CPU/Disk benchmarks on both servers. On the dedicated server you can just do cat /proc/cpuinfo but I don't know how that would work on the VPS.
• Compare the Apache configuration on both sites for any differences. Try using the identical configuration files on the new server if possible (back up the existing ones first).
• If using PHP look at the php.ini configuration (or similarly for other dynamic languages).
• Look in the error logs on the new server for any relevant messages (the Apache error log and /var/log/messages, /var/log/secure in particular).
• Double-check that you are never using swap memory when running your test scripts.
• Try reducing the size/scope of your test scripts until you find the smallest/shortest test that duplicates the slow speed issue.
• When all else fails double/triple-check all your assumptions (ex: scripts don't use much RAM/CPU).

Answer 2

Just a guess:

"The scripts don't fail, they just run slow. Sometimes they go faster at random.. but mostly just slow now."

Your test script sends 2500 xml requests to your server, times 4 as you describe (10k requests). Your FTP test proved that the bandwidth is not the problem (you've sent the 2500 XML files through a single connection, the transfer was done in a snap).

Could it be, that your site demands an amount of concurrent connections which wasn't expected/predicted by your current provider? And now, maybe, your site concurs with another popular site served on a server which sits in the same rack as yours?