Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have really strange problem with my servers. I have two servers, both with same processors, hard disks and everything else except RAM. First one have 16GB DDR3, the second one is with 32GB. On the first one the OS is Ubuntu server 10.04.4, on the second - Ubuntu server 11.04.

So .. here is the problem. I have relatively heavy php/mysql application running on both. On the first one I have 3 instances of the application, on the second just one, but the second server is much slower. (I`ve tripple checked the config files for apache, mysql and php) and they are absolutely the same. When I run ab on the first one from localhost:

ab -n 1000 -c 100 http://local_page/

Concurrency Level:      100
Time taken for tests:   5.056 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      26133000 bytes
HTML transferred:       25721000 bytes
Requests per second:    197.77 [#/sec] (mean)
Time per request:       505.642 [ms] (mean)
Time per request:       5.056 [ms] (mean, across all concurrent requests)
Transfer rate:          5047.15 [Kbytes/sec] received

On the second one the situation is different:

Concurrency Level:      100
Time taken for tests:   20.593 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      28437000 bytes
HTML transferred:       28024000 bytes
Requests per second:    48.56 [#/sec] (mean)
Time per request:       2059.296 [ms] (mean)
Time per request:       20.593 [ms] (mean, across all concurrent requests)
Transfer rate:          1348.54 [Kbytes/sec] received

From what I see the second one, which imo should perform better is much slower. I don't have any more ideas what to tweak to speed up the second server. I know it is shot in the dark, but could you point me to some general guides to check what is wrong ?

On the first server, when I have many users I have high cpu and ram usage (as expected), on the second one - even when many users are connected I have about 50% top cpu usage and about 50% used ram. Disk reads aren't intensive. Most used process is mysql, which on the second server rarely goes over 50% cpu usage and 5% ram usage acording to top. Maybe the best idea for now is to force mysql to use as much resources as it can.

share|improve this question
    
Do they host the exact same application, using the exact same configs, with the exact same dataset (typically in a database)? –  3molo Mar 21 '12 at 9:02
add comment

4 Answers

There's so many possibilities here it's hard to know where to begin. You're not even running the same OS, so the versions of everything is going to be different, along with the default configurations from which you're working.

Really, all I can suggest is to play hunt the bottleneck. I'd lean towards an untuned MySQL server, as that tends to be the first point of call since default MySQL configs are woefully underspecced for modern hardware. It's really hard to say for sure, though, since the number of potential variables is so huge.

share|improve this answer
    
Nice link. I think that might become required reading material for new hires. Extra points for the Wumpus references. –  Ladadadada Mar 21 '12 at 9:47
add comment

What I would recommend is starting Apache up in single-child mode and running strace on it with the -tt option. The command would look something like this:

sudo strace -f -tt -T -s 500 -o httpd.trace httpd -X

What you are looking for is a large gap between syscalls or a syscall that takes a long time.

Based on your talk of MySQL, I suspect it will be a poll() or multiple of them. If the problem turns out to be MySQL, you might be able to find out what's causing it by turning on the slow query log, setting the long_query_time to something low like 1 second or 0.1 seconds and turning on log_queries_not_using_indexes.

You can also find differences between the performance of servers by using monitoring and graphing tools like Munin, Cacti or Zabbix. I strongly recommend installing one if those.

share|improve this answer
add comment

50% CPU limit cap looks awfully like the PHP setting open_basedir

Without knowing what application you are running, I cannot suggest any more.

But bear in mind, such a short test will likely only be testing thread creation/spawning rather than the app performance itself. It also will be hitting RAM more than anything else due to disk buffers/cache, so if you've got a more optimal memory configuration in one server over another (eg. triple channel vs. single channel), it will have a HUGE impact.

Run a bootable version of memtest on each machine and it will give you a rough indication of memory bandwidth. If its much lower on one machine, there is your answer.

share|improve this answer
add comment

I would suggest you looking into the fact that Intel memory controllers scale frequency down as you fit memory slots inside the machine. As an example you have 3 branches for each memory channel (9 Dimms) if you fill only 1 branch the memory frequency will stay at 1.6Ghz if you fill all the dimms it will be lowered to 1.06Ghz. Check that out, just by removing half of the memory and keeping the same configurations on the secod machine. Often more doesn't mean better ;)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.