Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 1 down vote favorite

I have a SuSe Linux server that is hanging due to unknown issues, I'm to the point of wondering whether there is a runaway process being run every now and then that is causing it to hang.

The issue is that I don't know where to look for it after the hang happens... (I don't want to spend a whole day looking at top to find a runaway) -so my question is this: is there any log in SuSe that logs runaways? If not, is there a way to log such things?

Thanks a lot!

share|improve this question
It might be something totally different from what you suspect it to be. Describe the symptoms, describe when it happens, and describe what you suspect the server is doing just before the hang. – Nils May 18 '12 at 20:36
Thanks, I wasn't descriptive enough, I'm sorry. The hang I'm seeing is just that it gets unresponsive, I can't ssh to it, console through ILO is frozen too, when it happens I have to reboot the server. /var/log/messages just shows a bunch of failed mounts (not that much for it to hang) and the server hardware is an HP ProLiant BL460c G7 ... the HW is working OK.. that's why I'm getting to the point of thinking that its maybe due to a runaway that someone or something is running. Thanks a lot! – Dan V May 25 '12 at 22:10
Can you tell us what this server is doing? What type of application is running on it? Since this is ProLiant hardware, you have an Automatic Server Recovery watchdog timer that can be set in the BIOS. It may be able to trap the error and force a system reboot. – ewwhite May 25 '12 at 22:21
iLO freezes, too? Not sure about HP Proliant, but usually that is a small computer of its own. – Nils May 26 '12 at 20:10
Hi Nils, iLO doesnt freeze -but console through iLO gets unresponsive (I mean I can get to iLO, see the status of the server HW, etc. but the server console is frozen). @ewwhite ... I've never heard of ASR I'll dig into it thanks! – Dan V May 28 '12 at 17:22
show 1 more comment

3 Answers

up vote 2 down vote

More information would help. How are you defining "hang"? Assuming you have physical access to the server, you can check to see which kernel messages appear on screen following the hang. Does the server require a reboot after it stalls?

You can trace the normal system logs up to the point of the crash in /var/log/messages. If you have an open session when the server stalls, look at the driver messages by running dmesg.

Do you have any details on the hardware? If this is server-grade hardware, you may be able to check the system's hardware logs to see if there's an issue like bad RAM, etc.

share|improve this answer
The hang I'm seeing is just that it gets unresponsive, I cant ssh to it, console through ILO is frozen too, I have to reboot it when it happens. /var/log/messages just shows a bunch of failed mounts (not that much for it to hang) and yes it's a server HP ProLiant BL460c G7 ... the HW is working OK.. that's why I'm getting to the point of thinking that its maybe due to a runaway :s – Dan V May 25 '12 at 22:06
So it's HP hardware. Do you have the HP Management Agents installed for your OS? That will provide good insight into what's happening. – ewwhite May 25 '12 at 22:19
up vote 0 down vote

No there generally isn't a mechanism to tell you what exactly broke causing the "hang".

While your machine is running use top to look for processes eating too much CPU, free to check for memory problems (Swapping to disk can make a machine very very very slow) and look through /var/log skimming files to see if anything looks wrong.

ps aux | grep Z will filter out zombie processes if there are any.

share|improve this answer
The problem with the grep Z is that it will grep every single process with a Z on it however my issue is that when it happens I'm not able to run top nor free nor ps, the server gets frozen.. so the only thing I have left are the logs after I reboot it... probably I'll need to write a script of some sort to monitor for runaways or something similar... thanks anyways :) – Dan V May 25 '12 at 22:08
up vote 0 down vote

I got a case open with SuSE where a server freezes. They recommended these steps:

  • Connect a serial console (not very interesting for freezes), and you got iLO...
  • Redirect syslog to a remote-machine (so you can see "famous last words" - perhaps just before the originating system can sync it to /var/log/messages)
  • Install a KDUMP-Kernel and a debug-kernel (gives you the chance to get a kernel-oops rather than a freeze)

The last helped in my case - but I was able to recrate the problem by triggering a certain action - then I got a Kernel-Debug just before the freeze and with that SuSE was able to provide me a PTF (point-to-fix) Kernel, which removed the problem.

But still you did not describe under which circumstances your problem occurs - in the middle of the night? Never during work?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.