Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a problem with a Debian based Server, unfortunately housed in a remote location. Two days ago, it was suddenly down and completely unreachable. After having someone reboot it, it worked for about a day, then seemed to be down again.

This time, however, it was different:

-The server reacts to pings

-It is scannable, and all the relevant ports are open But:

-Open shells were immediately blocked, and no warning had been shown

-It's not possible to open any connection - the TCP/IP handshake seems to have some problems:

SSH handshake

As this is a private server of a small community, we don't have the resources to pay a technician to fix the problem for us, so any hints how to fix the problem after the next reboot would be highly appreciated...

share|improve this question

4 Answers 4

up vote 2 down vote accepted

It sounds like some process on the machine is going out of control and consuming so many resources the rest of the system can't function. (RAM, CPU, disk IO). (Or, as others point out, it could be a firewall issue. Or something on the network in between.)

But right now, you have no visibility into what's going on. You need that.

You should:

  • set up remote logging with syslog
  • set up a remote monitoring system of some sort

And, since the problem seems to be happening quickly and currently, you can take an ad hoc approach as well: run top remotely via ssh and leave that open. Watch what happens when the system dies next. But ultimately, you'll benefit from remote logging anyway, so might as well set that up.

share|improve this answer
1  
mattdm, it really was clogged up ram, and with your tips i was able to diagnose the faulty script fast enough - a runsv script in combination with an updated, but incompatible software package lead to the spawning of thousands of tasks... However I'm still amazed by the effect that this had :-/ –  Rnatau Jan 12 '11 at 8:59
    
It's possible to configure a Linux system to be more resistant to that kind of resource exhaustion, but it decreases flexibility (and possibly overall performance), so distributions are generally not configured that way out-of-the-box. –  mattdm Jan 12 '11 at 13:27

If you have out-of-band access to the server such as serial console or KVM; you should log in and diagnose the server from there.

To me it sounds like either the whole system is hanged (but ping is handled at a low level by the kernel, which is why it still works) or you have a firewall that is blocking you.

share|improve this answer

It looks like a problem with the firewall. 1. client-> Server syn 2. server-> client syn,ack 3. client-> ack

From the screen shot it appears that your ACK in step 3 are not being acknowledged by the server. I think you firewall doesn't allow related connections. You would need a similar rule in your iptables.

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

share|improve this answer

mattdm, it really was clogged up ram, and with your tips i was able to diagnose the faulty script fast enough - a runsv script in combination with an updated, but incompatible software package lead to the spawning of thousands of tasks...

However I'm still amazed by the effect that this had :-/

share|improve this answer
1  
Please post comments as comments, not answers. This is not a discussion forum. –  John Gardeniers Jan 11 '11 at 20:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.