Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I rent a dedicated server (with Intel Haswell CPU and custom hardware) at a lowcost hosting service and use it with CentOS 6.4 / 64 bit Linux (with stock kernel: 2.6.32-358.14.1.el6.x86_64).

Every few weeks it hangs and the other customers seem to have similar problems.

In the dmesg output I see (here is the full dmesg output):

CPU0: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz stepping 03
....
NMI watchdog enabled, takes one hw-pmu counter.
....
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
iTCO_wdt: Found a Lynx Point TCO device (Version=2, TCOBASE=0x1860)
iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)

and in the process list I see:

#  ps uawwwx|grep [w]atchdog
root         6  0.0  0.0      0     0 ?        S    Aug22   0:00 [watchdog/0]
root        10  0.0  0.0      0     0 ?        S    Aug22   0:00 [watchdog/1]
root        14  0.0  0.0      0     0 ?        S    Aug22   0:00 [watchdog/2]
root        18  0.0  0.0      0     0 ?        S    Aug22   0:00 [watchdog/3]
root        22  0.0  0.0      0     0 ?        S    Aug22   0:00 [watchdog/4]
root        26  0.0  0.0      0     0 ?        S    Aug22   0:00 [watchdog/5]
root        30  0.0  0.0      0     0 ?        S    Aug22   0:00 [watchdog/6]
root        34  0.0  0.0      0     0 ?        S    Aug22   0:00 [watchdog/7]

Does this mean, a hardware watchdog is already active at my server and will reboot my machine in under 30 seconds of being frozen?

(In the /etc/sysctl.conf I have put kernel.panic=10, so that it doesn't stuck in kdb console anymore).

Or do I have to install and start the CentOS package watchdog?

share|improve this question
3  
Why are you OK with this server hanging so frequently? Is it non-critical? –  SoFLy Sep 7 '13 at 13:13
    
Because the price is good (50 EUR for Haswell + 32 GB RAM) and I also have all my domains there... –  Alexander Farber Sep 7 '13 at 16:16
    
Not the answer you want to hear, but the correct answer for your situation is to ditch the crappy hosting company. Of course, if this is a Development environment, it's off-topic for Serverfault. Which I'm certainly willing to ignore as this looks generally interesting and useful anyway. –  Magellan Sep 7 '13 at 17:23
    
Instead of trying to mitigate this error by making it reboot on hang, you should probably just leave the host and find another one that doesn't have hanging machines, since it sounds like this could be an issue with the host. Or contact them? –  SoFLy Sep 8 '13 at 0:51
    
@SoFLy The OP says this has been discussed with the host via public forum. It still doesn't mean that the host is doing a good job. This is likely a bad hardware/driver/OS interaction. –  ewwhite Sep 8 '13 at 3:07

2 Answers 2

up vote 4 down vote accepted
+50

Linux has a generic watchdog interface. You can use it by either enabling the NMI watchdog your iTCO_wdt hardware supports or by installing and configuring a software watchdog which does not depend on the hardware.

share|improve this answer
    
Thanks. Will the "wathdog" CentOS package use the "iTCO_wdt" thingie seen in my dmesg or are they unrelated? –  Alexander Farber Sep 8 '13 at 7:03
    
I edited my answer to make it more clear that they're unrelated. –  sciurus Sep 8 '13 at 21:29
    
But the hardware support for this particular watchdog is not available in the kernel the OP is using. If it's a hardware/driver interaction, the software watchdog may not be the answer. –  ewwhite Sep 9 '13 at 14:41
    
@ewwhite how do you know their kernel doesn't support the hardware watchdog? I think their dmesg output shows it's supported. –  sciurus Sep 9 '13 at 20:48
    
The code for that Intel chipset was not backported into their kernel. It is in the CentOSPlus variant of the kernel. See the kernel changelog or: [watchdog] iTCO_wdt: add Intel Lynx Point DeviceIDs (John Villalovos) [738470] –  ewwhite Sep 9 '13 at 21:17

Well, there are a few issues to tackle here...

  • What happens when the server hangs? What's on the screen? What's in the logs? Do you have to engage with the hosting provider to reboot? Can you perform the reset on your own?

  • Your server should not be hanging, stalling or crashing!! Having worked in environments where low-end, DIY or custom hardware is used, I understand that the service provider's aim is to cut costs. However, if there's a stability issue, the onus is on the provider to remediate those issues. It's not difficult to build a stable Linux server platform. Yet, it happens more often than it should. If the combination of hardware/software/OS/firmware is toxic, that's a bad sign. The provider should be operating at a scale where they should be able to understand problems before they impact multiple clients.

  • Does your hardware have an IPMI device? Do YOU have IPMI access? Typically, watchdogs are part of your out-of-band management device. For instance, HP ProLiant servers have their Automatic Server Recovery (ASR) feature set to handle this.

  • The device your system detects is part of the Intel chipset in use. So there is technically a watchdog device and there is generic kernel support for it (it looks like it's in the CentOSPlus kernel, not the one you have). However, the watchdog package can help as a software-level watchdog, outside of the hardware hooks you may have.

But again, you're treating the symptom here. It's important to get to the root cause. If other customers are encountering these issues, you all need to resolve this with the service provider.

share|improve this answer
    
On the screen is a useless kernel stack trace. This all been discussed en masse at the hoster's forum... I don't have an IPMI access, because it costs extra money... I am sorry, but your answer is useless for me. –  Alexander Farber Sep 7 '13 at 16:23
    
Well, think of this as a hosting problem. Likely driver or hardware related. You have a watchdog on your server hardware, but the installed Linux kernel can't take advantage of it. The support was built into the CentOS Plus kernel, though. This was backported from newer kernel revisions. You can install the watchdog app as well. But the point of this answer was to see what options you had available to you. E.g. IPMI is a very good start. –  ewwhite Sep 7 '13 at 16:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.