Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have 4 RHEL 5.3 servers (running on HP blades) on my server farm that seem to hang every 2 weeks and needs rebooting to resolve the problem When this happens the server goes unresponsive to ssh attempted connections and cannot be reached.It can only be reached via the ILO. before the server goes unresponsive ..sometimes the ssh session hangs for a long long time before the server goes into unresponsive mode.

After cold rebooting via ILO it comes back to normal operation mode .I have looked through the log files thoroughly and found nothing .The other RHEL 5.3 server on this farm dont seem to have that problem and dont have IP tables enabled on them

The only hting i found is that these affected servers have IPtables enabled on them and seem to have high packet rejections..thats all i seem to see in their logs ie system logs /var/log/messages.High volumes of packet rejections in log files because IP tables is turned on

Could it be the IPtables causing this ?.the logs show no indication of any disk ,hardware problem or any other problem. Patching is not an option at the moment.If it IP tables causing this can anyone explain please Help Please anyone any help would be greatly appreciated

share|improve this question

4 Answers 4

These are HP servers, so do you have the HP Management Agents running? What is the ASR watchdog timer value set at? I'm assuming the timeout is the default 10 minutes. Are you seeing anything in the ILO log or the system's IML log? How long are the systems stalled before you reboot them?

I would look in the ILO and server's IML logs for info. You may have a hardware issue or it could be something triggered at the application/OS level.

share|improve this answer

High levels of packet rejections can happen for any number of reasons -- it usually means that iptables is doing its job. After all, what good is a firewall that doesn't block any packets, right?

You're asking for a very specific answer to a very general question.

What you're asking could be a reliability problem in some service on the system, or it could be a performance problem. There's no way to check until you start checking logs and performance counters. (You do have performance metrics being logged somehow, don't you?)

Can you answer the following questions about each outage?

  1. What was the exact time, to within 5 minutes, that the server went dead?
  2. Does syslog show any pertinent errors from around that time?
  3. What was the load average leading in each ten-minute interval in the hour leading up to that time?
  4. What was the memory usage in each ten-minute interval in the hour leading up to that time?
  5. What was your swap in/out rate in each ten-minute interval in the hour leading up to that time?

If you don't know the answer to any of these, you need to look at your data collection methodology until you can. If you're not sure where to begin tracking performance metrics on Linux, sar is a good starting point. You can also look at Performance Co-Pilot, Munin, or any number of other tools.

Afterwards, if you still think iptables is to blame, you can turn on logging by adding something like the following to your iptables config:

-j LOG --log-prefix="" --log-level=info

Hope this helps.

share|improve this answer

It's possible that the iptables rules may be contributing / causing the problem, particularly if they are stateful - but it's not simply having the iptables module loaded.

But you don't say what the iptables rules are.

share|improve this answer

As symcbean said, it matters more what your iptables rules are. Make sure that you aren't logging dropped packets unless you have a specific need. Also check your selinux logs (/var/log/audit/audit.log). I've found that half of my problems in RHEL stem from selinux. Make sure your log partition isn't filling up.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.