3

I have a quad hexacore system based on the Opteron 8345 chips - that's 24 cores in all with 48GB of ram. This box was meant to replace the current six Intel core box which my current LAMP stack runs on. I am however getting even worse performance than from the six core. I'm running:

  • Centos 5
  • MySQL 5.0
  • PHP 5.2.13
  • Apache 2.2.9

It's almost as if the box is too powerful? Has anyone had any experience with large SMP boxes like this? 

[root@gb-sanguine ~]# uname -a
Linux gb-sanguine.bountyonline.local 2.6.18-194.3.1.el5 #1 SMP Thu May 13 13:08:30 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@gb-sanguine ~]# php -v
PHP 5.2.13 (cli) (built: Jun  2 2010 16:29:01)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies

[root@gb-sanguine ~]# httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

[root@gb-sanguine ~]# httpd -v
Server version: Apache/2.2.15 (Unix)
Server built:   Jun  2 2010 11:15:28

[root@gb-sanguine ~]# mysql -V
mysql  Ver 14.12 Distrib 5.0.77, for redhat-linux-gnu (x86_64) using readline 5.1

Please see http://pastebin.com/raw.php?i=KSWxqmJe for dmesg output.

Many thanks in advance for your help folks.

Improve this question
1
  • 1
    Out of curiosity what is the RAM config on this box vs it's predecessor and be precise - give us the no of DIMMS, size, ranks of this system and the one you are comparing it to (and the no and model of CPU too). Details can matter quite a lot here. It certainly should perform better but a well optimised 6 core architecture vs an unoptimised 24 core could easily make 6 > 24.
    – Helvick
    Jul 3, 2010 at 1:11

6 Answers 6

Reset to default
4

cat /proc/cpuinfo

are all CPUs actually recognized? I don't know about CentOS, but, some kernels turn on SMP but limit the number of cores to 8 as a default.

Does top show processes on each cpu? shift-O, j

Is your application not multithreaded?

-- It does recognize all 24 cores, so, that theory is effectively negated. The only other simple thing I can see is that your cpu powersave governor MIGHT be running the cpus at 800mhz rather than the 2.6 ghz.

What does your application do? typical webapp? or is there a part that is not able to be multithreaded? Is mysql configured to use more cores (thread_concurrency)? When you watch the machine running, what is slower? network connection time? CPU throughput? disk access? sysctl.conf settings migrated over from the other box? any tweaks/modifications to mysql/apache/php configs that didn't get moved over?

can you paste: cat /proc/mtrr

Improve this answer
1
  • I have reduced to 12 cores now at get the following: gb-sangunie:~# cat /proc/mtrr reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1 reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1 reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1
    – Khushil
    Jul 4, 2010 at 0:48
2

Cache memory issues. You need to be careful about how those cores are socketed and how the tasks spawn across them. I had a server go from fast 6-core to slow 8-core in a 12-core (2-socket) system. So, just having lots of cores and RAM does not mean that you should just blindly make use of them all.

I would suggest using your 24-core system to run a virtualisation host like KVM. Then, it may be easier to partition your workload across a number of smaller VMs. You can pin specific cores to specific VMs. So, if your 24-core system is a 4-socketed 6-core or a 6-socketed 4-core, you can have fine-grained control over the performance. There will be a difference in performance between the two.

Cheers.

Improve this answer
1

Put a newer MySQL on it - 5.1 at least, or better, Percona's XtraDB. Make sure you're using InnoDB. MySQL 5.0 scales negatively (i.e. performance decreases) with more than 8 cores in my experiences.

Improve this answer
1

Depending on the workload and kernel version, increasing the number of CPU may very well decrease performance (semaphore implementation of the Big Kernel Lock with heavy POSIX file locking usage comes to mind, this is something that has been observed in RHEL/CentOS 5).

Have you checked whether you get better performance by reducing the number of CPUs? (maxcpus=)

Otherwise, you should profile your application/system to track down the bottleneck. If this is CPU-bound, OProfile should help. SystemTap also comes to mind.

Improve this answer
1
  • I have now switched to Debian Lenny. This is a standard LAMP stack running a Drupal website. I now have mysql 5.1 installed.
    – Khushil
    Jul 4, 2010 at 0:49
0

Did you see CPU performance bottlenecks in the past? You be seeing a throughput issue now, instead of a CPU bottleneck.

With 6 sockets, the data needs to get from RAM to each CPU, and depending on how the cache is managed, you may be doing a lot more Cache misses (L1, L2, L3 cache in this context, that is usually on chip, or at least on-die in multi core instances) and be doing memory reads instead of L2 cache reads.

Sometimes throwing resources at a problem does not help, if the problem was not a lack of resources.

However +1 for kick ass hardware! Wow! 24 cores!

Improve this answer
2
  • The problem with the old box was always the CPU bottleneck. I was hoping for a rather big boost with this box. The more I think of the problem the more I come to realise that we may just have killed performance by performance? :-\
    – Khushil
    Jul 2, 2010 at 20:17
  • 3
    @MutabiMan What exactly was your CPU Bottleneck before? Speed of a Single CPU or total throughput? If you had a 2.8 or 3.0 GHz CPU before and now a 2.6 GHz one, it potentially slower for Single-Threaded Apps but you can now run more of them at the same time.
    – Michael Stum
    Jul 2, 2010 at 20:42
0

Actually, as a simple LAMP stack server, it is a bit too powerful.

You can look into virtualize the box and it will replace not only what it is intended to replace and maybe some more servers in your farm.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .