Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Ok, our new build is having 100% cpu spikes on each server at random intervals. For long durations it make the site totally unresponsive - this will be at peak times as people in different countries log on to the site etc.

We've looked at perfmom, memory profilers, CLR profiler, sql profilers, Red gate ants profiler, tried load testing in UAT - but cannot even reproduce the problem. This could mean only thousands of users hitting the live site causes it to happen.

One pattern we did notice was that the new code - the broken build - actually uses noticably less threads.

We are also using spring for IOC - does this have a bed reputation?

To make things worse, we cannot deploy to live due to the business impact - so cannot narrow the problem down to subset of the new features we've added.

We truly are destroyed - has anyone got any battle scars that may save us a few lives?

share|improve this question

migrated from stackoverflow.com Nov 24 '11 at 15:44

This question came from our site for professional and enthusiast programmers.

    
What do the temperature sensors report? I wonder if your power supply can't keep up. (No idea how to check this.) –  sarnold Nov 23 '11 at 12:10
2  
When you say brings the server down can you add more detail, is it BSOD? Do you mean it restarts or maybe an app domain restart. –  RubbleFord Nov 23 '11 at 12:13
    
There is no way at all a "100% cpu spike" could "bring down" the server. It would have to be pegged at 100% for quite a long while, coupled with trouble with heat dissipation. –  Andrew Barber Nov 23 '11 at 12:16
1  
What is it doing?? Which process is using the CPU at the peak? This is the most important question. –  Aliostad Nov 23 '11 at 12:17
    
Updated my question - is this better? Thanks for the -1 :) –  user880954 Nov 23 '11 at 12:28
show 6 more comments

5 Answers 5

I suggest doing memory dumps and analyzing them in WinDdg with Sos. I fixed some problems on our production I probably wouldn't be able to diagnose without WinDbg.

Tess Fernandez has great blog where you can learn how to analyze memory dumps.

share|improve this answer
    
that blog is an excellent resource and we have been using it. Our problem is we can't recreate the problem again and get the dumps. –  user880954 Nov 24 '11 at 12:34
1  
To recreate the problem, you may hammer your test system with jmeter (jmeter.apache.org) and ab (httpd.apache.org/docs/2.0/programs/ab.html). With these, multicores, a fast LAN and some colleagues, you should be able to stress the server enough. –  Roman Nov 24 '11 at 16:05
add comment

This is typically caused by large long-lived object cleanup in the GC(stackoverflow had this problem, see link). Are you storing lots of object collections in cache or session?

Assault by GC

I also recommend you build and configure a new server in production to test. If you have random craziness and don't know why and can't reproduce it, I'd point the finger to hardware or configuration, not code.

share|improve this answer
    
We can't put any new code live because it adds news features. When the code was live, the GC usage was the same - including for generation 2. Thanks though - do yo have any more suggestions? –  user880954 Nov 23 '11 at 12:49
    
It's not impossible, but the hardware and configuration are nearly the same as the last deploy which we have reverted back to and is working successfully. –  user880954 Nov 23 '11 at 18:36
add comment

Is this a virtual server with shared resources or a physical server? If it is the former perhaps you could look at dedicating resources to this server. Good luck...

share|improve this answer
    
Physical servers unfortunately. Thanks –  user880954 Nov 24 '11 at 12:35
add comment

Try using a cache server as a frontend like Apache Traffic Server (ATS).

While this will not resolve the problem, it may help to identify it because you will at the same time move the potentially harmful load from the backend (seeing if the frontend also has problems) and make things less heated on the backend so it will be easier to see what's wrong.

share|improve this answer
add comment

Trying to guess the fault without the data is pointless. Yes someone on stackoverflow or in your engineering team might get lucky but that's just bad engineering, and you can't put a plan on how long it will take you to try every guess, and if thy would even find the problem.

  1. You have to repro the problem. Jmeter is a good start because of its breadth, but we can't recommend the right tool without knowing our architecture.
  2. Logging specially in your application layer is a must. You can enable IIS traces for slow performance, but the muppets at Microsoft made it so you can't capture the entire pipeline flow when it's slow. If it is so difficult to repro, you'd really like some logs to help you narrow down where the problem is. (like oh, it's whenever we call this stored proc).

The 100% CPU is a little suspicious in the sense that it's unlikely to be I/O (providing the db is another box, a slow database should not cause 100% CPU on the webservers). You need to look closer to home.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.