Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have Ubuntu 9.10 dedicated server (un-managed) and it started dying few weeks ago.

Before i request hardware inspection i would like to confirm that there is no software issue of some kind going on on the server.

Server is unmanaged so I need to do everything by my self.

Server is hosting few WP sites and one VBulettin forum.

Here is my php info http://pastebin.com/hSQVQBMR

Server has worked * flawlessly * for about a year, in the mean time, no a single restart, and now it started to suddenly hang.

It always happen at approximately same time (4-6 am CET) when we have most visitors online.

But strange thing is that this never happened before, it worked very well for a year or more.

So my question is - how to investigate?

I have cacti set up from day one - and there is no unusual activity what so ever.. Further more, every time it hangs out it happen on down slope of load and mysql queries chats (and all other load-related charts)

What I didn't have was number of sockets chart, but I added that today.

Thing that worries me the most is that every time i requested restart (aprox 4 times in last 7 days) support guy told me that he was getting black screen (so i guess this is not the case of load ~50)

What log files should I watch?

What entries in those files should I look for?

share|improve this question
3  
Can you be more specific about what "dying" means to you? -- Are we talking just the web server, or the whole box? Can you still SSH in? Does the box panic? See also serverfault.com/questions/127352/… Re: diagnosing mysterious problems -- Bad RAM is my first guess in situations like this (and yes, RAM can go bad a year or more after it's installed) –  voretaq7 Aug 19 '11 at 16:43
    
Whole server dyes. Not I or support guy can log in, and it has to be restarted manually. –  kodisha Aug 19 '11 at 16:51
2  
OK, I'd start with the memory tests in the question I linked then. Sounds like a hard crash/kernel panic, which is often bad RAM –  voretaq7 Aug 19 '11 at 17:49
add comment

3 Answers

Look for Memory errors and HDD errors in /var/logs/messages to start with.

Is this server in a data center? With proper electricity feed? Variation in electricity can cause a server to crash and can also prevent it from booting if not enough power is available.

You can also test your hardware, especially your Memory and HDD.

share|improve this answer
add comment

Set up CPU temperature monitoring, if you haven't already. If the problem is overheating then you may be able to see a sharp rise in temperature just before failure.

/var/log/kern.log would be worth a look. However if the system is crashing it may well be unable to write anything to it when it really matters.

If you can get access to the console - or perhaps better, use a serial console and leave something logging everything written to it (I use 'screen' for this) - then you may be able to see what the kernel says when it crashes.

share|improve this answer
add comment

Does it "die" when there is nothing to do for a while? Then power-saving may be the problem here. Try to disable it completely or at least prevent it from switching a CPU or core into C-sleep-state.

I`ve got a bunch of Dell Servers that expose the strangest errors if C-state is enabled in BIOS power savings.

Do you know what kind of hardware is being used (make, model, CPU - propably Intel)?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.