I have a 24/7 system with a couple of semi autonomous nodes (embedded x86 minipc) running Ubuntu jaunty (9.04). Each of them need network connection to gather information to operate. I use monit to restart some services if they're down for some reason and I monitor each node using Nagios 3, but I don't know a good way to evaluate (automatically) system sanity under Linux. To be more specific, in case the network connection is having problems (e.g. the network driver isn't working properly), how can each node evaluate its "health" to determinate that it needs a reboot (sorry for not being more specific)? Do you people have opinions/experience about it?
Thanks in advance!