Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have 2 Debian Linux 6.0.4 servers that have a strange behaviour: after 5-7-10 days they hang. By this I mean the servers need to be restarted and before that ping won't answer.

I've been struggling with this problem for a couple of months now and here's some thoughts/what I tried without being able to solve the problem.

  • I changed the RAM on a server. Being 2 different servers I doubt that it could be something related to hardware as a 3rd identical server won't have this problem.
  • I logged the server load and when it crashes the load is fine (quite low)
  • I cannot find anything in the server logs, logs are fine till the server freezes.
  • I don't have access to console unfortunately.

While I have years of admin experience I have never encountered such an issue and right now I have no idea where else to investigate.

If you have an idea of what I could try in order to fix the problem please share it with me:-)

share|improve this question
    
What does demsg say? Also were these two installed from the same cd? It might be a problem there. (just guessing) –  Lucas Kauffman Jun 26 '12 at 16:17
1  
Broadcom NICs involved? –  Mike Pennington Jun 26 '12 at 16:33
1  
If you don't have access to physical console, but the servers share LAN you may want to set up netconsole: kernel.org/doc/Documentation/networking/netconsole.txt –  Hubert Kario Jun 26 '12 at 16:51
    
@LucasKauffman: I installed it via network from the same image. The servers are rented from hetzner.de –  Alex Flo Jun 27 '12 at 7:03
    
@LucasKauffman dmesg just stops recording anything suddenly then it starts recording again when the server boots up. –  Alex Flo Jun 27 '12 at 7:04
add comment

3 Answers

Do the servers really hang or are they just unreachable by ping?

Install a monitoring tool such as Munin (or similar) which will show you graphs of not just the CPU load but also memory stats, disk usage, and various other bits and pieces - you can configure it to monitor lots of aspects. Nex time the server hangs, check the graphs for any unusual signs. You will learn to see what a normal graph looks like so anything out of the ordinary is suspicious (although not necessarily wrong).

Are you sure you are checking all server logs? ie do you have web/mail/ftp/dns/other servers? check all such logs! Don't forget to enable debug logging while troubleshooting.

If the server crashes every week or so it could be something that happens regularly, ie a cron job, or log rotation, stuff like that.

Make sure you get all system emails (root alias). You can even install OSSEC which is a great tool for keeping an eye on the logs and getting emails when things go wrong. But this OSSEC tool is just an automated way of looking at the logs, so nothing magical.

Networking issues? dhcp lease expired?

share|improve this answer
    
I believe the server hangs because I cannot see anymore messages logged besides not responding to ping. –  Alex Flo Jun 27 '12 at 7:10
    
I looked at (all) the logs and they looked like the server was plugged out of the socket (though this isn't the case I believe:) –  Alex Flo Jun 27 '12 at 7:12
    
dhcp if that was the case then I would have seen something in logs I believe, not to mention that even if the server was not accessible because of a networking issue my loggings should have worked just fine which is not the case. –  Alex Flo Jun 27 '12 at 7:14
add comment

Please show relevant content of /var/log/messages and/or /var/log/kern.log it's possible the kernel logged some crash reports or something else that could shed some light. When I experienced such unexplained hangs it was due to a bad driver, because logging isn't very verbose I wasn't able to find out the exact driver.

In my case there were soft lockups (kernel: [XXXX] BUG: soft lockup - CPU#X). After some research I found http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=556030 and the last comment provided some insight and a way to make logging more verbose. It's an easy kernel modification but if you don't feel comfortable compiling your own kernel it may not be the best thing to do.

Just updating the kernel or installing a newer version and rebooting may fix the problem.

Quoting:

We extensively researched the problem.

The TLB flush softlockup is only a CONSEQUENCE of a deadlock.

Background: The TLB flush is issued by a CPU to a number of other CPUs using inter-processor interupts to progagate paging changes. Then the issuing CPU loops until all processor acknowledge the change. If such processor is in deadlock on a spinlock, this never hapens, then the softlockup triggers. The deadlock arise on a spinlock, this lock may be held by user code sometimes (through /proc or /sys interfaces of modules).

The only way to identify the root cause (i.e. which driver is causing problems) is to dump ALL CPU stacks in the soft lockup code.

One way to do that is to modifiy the kernel and add

            arch_trigger_all_cpu_backtrace() 

in the

            kernel/softlockup.c:softlockup_tick() 

function.

This is based on NMI IPI which ensure all stacks are dump, even in the case of deadlock (well don't expect the impossible to happen either).

You should easily find the faulty driver and post the relevant bug.

share|improve this answer
    
thanks for this suggestion but I have a 3rd server with exactly the same HW and the very same Debian version (same kernel version as well) and that server doesn't have this issue so I tend to believe its not a kernel problem. –  Alex Flo Jun 27 '12 at 7:19
    
anyway, I somehow believe it could be related to some Python scripts that I run on these 2 servers which hang: they could exhaust the maximum number of file descriptors. Would that be possible? And if so would ping stop responding? –  Alex Flo Jun 27 '12 at 7:20
add comment
up vote 0 down vote accepted

Apparently the problem was related to some python scripts that caused the server to hang. I don't understand why they hanged the server but at least they don't hang it any more.

share|improve this answer
    
Great, glad you resolved it. Be sure to mark this answer as "accepted" when you are able. –  jscott Jul 10 '12 at 2:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.