Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

===SOLVED===

This issue was solved. Turns out that ImageMagick has trouble with multiple CPUs. Compiling ImageMagick to use one CPU solved the problem.

================

I added a new web server as an upgrade but it falls over within seconds.

The old box has 8 Xeon cores at 2.33GHz. The new machine has 16 Xeon cores at 2.40GHz. Memory is 8G and 32G on the new machine.

The other major difference is a leap from 32 bit to 64 bit.

OS is CentOS 5.6 on both and Apache is 2.2.3-45 on both as well.

PHP is 5.2.10 and compiled by hand. configure options are identical except for the architecture bits.

From all of this info, you would think the new machine would scream but the current box handles the load and falls over occasionally. The new machine dies every time in less than a minute.

Memory is fine, I/O is good, but CPU is pegged hard. Here's the output from mpstat from both

old box

09:14:18 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
09:14:20 PM  all   31.34    0.00    2.62    9.68    0.12    1.00    0.00   55.24  11163.50
09:14:20 PM    0   53.00    0.00    5.50   16.00    0.50    6.50    0.00   18.50  10249.50
09:14:20 PM    1   36.68    0.00    2.51   11.06    0.00    0.00    0.00   49.75    126.00
09:14:20 PM    2   17.41    0.00    1.99    7.96    0.00    0.00    0.00   72.64    125.50
09:14:20 PM    3   41.00    0.00    3.00    9.00    0.00    0.00    0.00   47.00    125.50
09:14:20 PM    4   30.00    0.00    2.00    7.50    0.00    0.50    0.00   60.00    143.00
09:14:20 PM    5   28.50    0.00    2.00   12.00    0.00    0.00    0.00   57.50    142.50
09:14:20 PM    6   22.61    0.00    1.51    7.54    0.00    0.00    0.00   68.34    125.50
09:14:20 PM    7   21.50    0.00    2.50    6.50    0.00    0.00    0.00   69.50    125.50

new box

09:13:41 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
09:13:43 PM  all   98.69    0.00    0.81    0.00    0.03    0.47    0.00    0.00   4723.50
09:13:43 PM    0  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   1000.50
09:13:43 PM    1  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM    2  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM    3  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM    4   98.01    0.00    1.49    0.00    0.00    0.50    0.00    0.00      0.00
09:13:43 PM    5  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM    6  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM    7   98.51    0.00    1.49    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM    8  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM    9   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM   10  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM   11  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM   12   95.50    0.00    4.00    0.00    0.00    0.50    0.00    0.00     84.50
09:13:43 PM   13  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM   14  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
09:13:43 PM   15   87.56    0.00    4.98    0.00    0.50    6.97    0.00    0.00    3640.0  

Traffic comes in through a load balancer and is split 50/50 between the two. As soon as I turn on the new machine, load spikes to 200 and I have to turn it off as it stops taking requests.

strace against httpd doesn't seem that revealing but here's the output from an strace -c -f -p

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 73.52    2.763912         419      6594      1663 futex
  8.65    0.325110          55      5869      4099 open
  5.35    0.201250         107      1873       381 stat
  3.12    0.117305          67      1748       165 lstat
  2.30    0.086434        2010        43           wait4
  1.64    0.061543           7      8825       769 read
  1.31    0.049158         125       394           clone
  0.77    0.028874          53       543           chdir
  0.75    0.028356          29       973           munmap
  0.34    0.012783          35       370           times
  0.30    0.011298         257        44           madvise
  0.24    0.008897           7      1312           fstat
  0.22    0.008225           1      9341         2 poll
  0.18    0.006682           2      2777        14 write
  0.14    0.005358           5      1184           mmap
  0.13    0.005020          19       262           set_robust_list
  0.13    0.004990           3      1688        30 writev
  0.13    0.004799           7       671       598 access
  0.08    0.003194           0      6531           recvfrom
  0.06    0.002404           4       673         8 sendto
  0.06    0.002398           4       578           getcwd
  0.06    0.002367           5       491           mprotect
  0.05    0.002013           4       457           brk
  0.05    0.001965           2       883           semop
  0.05    0.001924           3       760           lseek
  0.04    0.001622           2       845           setitimer
  0.04    0.001525           4       412           epoll_wait
  0.04    0.001486           1      2595           close
  0.04    0.001430           3       412           accept
  0.04    0.001429           3       433       231 connect
  0.04    0.001388           1      1185           rt_sigaction
  0.03    0.000999           2       594           rt_sigprocmask
  0.03    0.000963           0      2325           fcntl
  0.02    0.000935           1       690           setsockopt
  0.01    0.000393           1       534           socket
  0.01    0.000380           1       393        12 shutdown
  0.00    0.000158           1       127           setuid
  0.00    0.000156           0       411           getsockname
  0.00    0.000156           2        70        46 unlink
  0.00    0.000080           0       254           epoll_ctl
  0.00    0.000000           0        64           ioctl
  0.00    0.000000           0        38         6 select
  0.00    0.000000           0        10           alarm
  0.00    0.000000           0       230           getsockopt
  0.00    0.000000           0         3           rename
  0.00    0.000000           0        22           getrusage
  0.00    0.000000           0       127           setgid
  0.00    0.000000           0       254           geteuid
  0.00    0.000000           0       127           setgroups
  0.00    0.000000           0       127           epoll_create
------ ----------- ----------- --------- --------- ----------------
100.00    3.759359                 67166      8024 total

========== EDIT / UPDATE ==========

I found that when I limited traffic to 10% on the load balancer as suggested, it still crumbled. When I beat on it with siege and 400 connections, it held up really nicely. Load increased but hovered around 6 and served all requests.

I have access logs disabled but I enabled for a bit and told the load balancer to start sending traffic again. I let this run until load hit 200 which was about 3 minutes and saved the log.

I parsed the log for requests to use with siege. This would give me a more accurate benchmark.

Sure enough, with no live data but just me hitting it, I spiked load to 200. I started chopping the file in half and testing top and bottom half. I'm continuing this until I can find the specific request or requests that break the server.

So far it's looking like stuff that makes heavy use of ImageMagick but I've whittled down 10K GET requests to 50 and still going.

share|improve this question
    
Just for clarification: Are you using mod_php? Are you using an opcode cache? Are the httpd processes what is using the CPU? –  sciurus May 3 '11 at 2:06
    
Using mod_php and APC for caching for both machines. Oh, and php.ini are identical. –  Seth May 3 '11 at 2:21

2 Answers 2

up vote 6 down vote accepted

This is not going to read so much as an answer but as a series of steps to help you track this down.

  1. Install Ganglia: http://ganglia.sourceforge.net/. It is my preferred load troubleshooting tool. You will like it.
  2. See if you can use your load-balancer to send a smaller percentage of the traffic, say 5%, to your new server. This will let your server hopefully stay up longer.
  3. Run top on your new server and sort by CPU via the "P" sort key. Look at what is taking the majority of the cycles.
  4. Double check all the PHP bindings to MySQL and Apache and libraries are installed correctly. This is my number one suspect for what is going wrong based on the information you've provided to date. The fact that you've hand compiled your PHP also raises a potential red flag. Double check your config options and make sure nothing has changed in what is expected or required across the 32/64 bit change.
  5. Enable and look at the logs: php error_log and apache error logs to see what is happening. This is always informative but you'll need to separate the signal from the noise in your particular environment.

One or more of these steps are likely to help flesh out what is going wrong.

share|improve this answer
    
Thanks. I did limit the traffic and it still crashed. I don't think it's load but that the code somehow puts Apache in a weird loop. Even with 5 people hitting the site at one time, it falls over. –  Seth May 3 '11 at 2:22
    
What does top show prior to the system hanging/crashing when you sort with key "P" ? –  dmourati May 3 '11 at 5:33
    
Can you also post the entire ./configure line you passed to compile PHP? –  dmourati May 3 '11 at 21:22

If you switch to your distro's php or php53 packages, does the high CPU usage still occur?

If you're using an opcode cache, does the high CPU usage still occur if you disable it?

If you run a tool like apachebench against a simple "hello world" PHP script instead of your actual application, does the high CPU usage still occur?

Can you disable PHP modules one by one and test if the high CPU usage still occurs after each one is disabled?

Can you use a PHP profiler/debugger to analyze what is happening in your application?

share|improve this answer
    
5.3 won't work with the code base so I'm unable to test. I've disabled APC with the same result. I've banged on it with siege(like ab) against the live site and it didn't take it down even when emulating twice the expected traffic. I did find one thing out which I'm going to post in an edit to the main post. –  Seth May 3 '11 at 3:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.