Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a 24/7 system with a couple of semi autonomous nodes (embedded x86 minipc) running Ubuntu jaunty (9.04). Each of them need network connection to gather information to operate. I use monit to restart some services if they're down for some reason and I monitor each node using Nagios 3, but I don't know a good way to evaluate (automatically) system sanity under Linux. To be more specific, in case the network connection is having problems (e.g. the network driver isn't working properly), how can each node evaluate its "health" to determinate that it needs a reboot (sorry for not being more specific)? Do you people have opinions/experience about it?

Thanks in advance!

share|improve this question
add comment

migrated from stackoverflow.com Dec 29 '09 at 17:41

This question came from our site for professional and enthusiast programmers.

5 Answers

I don't know of a situation when an automatic reboot is necessary and can be launched from the machine itself. In the worst case, you can set a watchdog that will reboot the machine if it's stuck. In most situations though, it is preferred to just restart services. If you want an intelligent way of doing that, I'd use puppet to manage dependencies between files, packages and services.

share|improve this answer
    
Agreed, automatic reboots smells like a way to fix the symptoms of a problem temporarily, rather than the underlying issue. –  rodjek Dec 29 '09 at 23:29
add comment

Do you people have opinions/experience about it?

I think you're anticipating and toying with black magick that is commonly associated with Windows.

I've never seen and would be very suspicious of connectivity issue that can be reliably fixed by rebooting. Even if it were to provide a temporary fix, I'd want to be pretty sure of the cause and resolution before bringing the machine back into service.

share|improve this answer
add comment

Do function testing (you can write Nagios checks for that, if there aren't anyone available, it is not that hard if you now some scripting language). Test that your services are reachable and functioning correctly from the Nagios machine.

The node itself can try to reach your Nagios machine and if it's unreachable just restart itself, but it's probably more preferred to run on hardware that have good drivers available in the first place...

share|improve this answer
add comment
  1. Use a hardware watchdog to monitor and reset the systems if they hang.
  2. Whatever the machines are doing, use monit or nagois to monitor how many requests/second or minute are performed and warn a human if that number drops below a certain threshold.
share|improve this answer
add comment

What about just bringing the interface down and then back up? It does fix most issues that rebooting would fix.

And just do it from cron, or use a script to check connectivity, and if things are bad, interface down and up, if that doesn't fix it, reboot.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.