linux server constantly kicking me out

Question

I have a linux server that has started acting a little funny...

First of all the box is way under utilized. It's a database server, with a very low load. I'll put vmstat below and top...

1. I'm running centos 5.4
2. This server has been running fine for weeks
3. It's firewalled, it's unlikely it's been hacked (chkrootkit agrees).
4. Load is really low.
5. I'm on a Gigabit switch with the server.
6. I'm not having this problem with any other server.
7. I have remote syslog set up on another box and it's not indicating any corresponding errors.

So, when I ssh(putty) to the box, if I haven't gotten on it in a while, putty immediatly throws a 'network connection reset' error.

Randomly it just kicks me out (or network drops)

a steady stream of pings shows now interruption.

--- vdbsrv1 ping statistics ---
80620 packets transmitted, 80619 received, 0% packet loss, time 23838ms
rtt min/avg/max/mdev = 0.234/0.273/1.741/0.043 ms, ipg/ewma 0.295/0.297 ms

SQL queries ("Select 1") to the box take about 5 seconds, even though SQL is reporting something like a .04 second execution time, so the other 4.96 seconds is a mystery

Anyone have any ideas?

top - 11:49:28 up 12 days, 17:05,  4 users,  load average: 0.00, 0.03, 0.01
Tasks: 150 total,   1 running, 149 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 99.3%id,  0.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8174028k total,  8074924k used,    99104k free,  4558300k buffers
Swap: 19464184k total,        4k used, 19464180k free,  2306952k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    1 root      15   0 10348  692  576 S  0.0  0.0   0:02.13 init

root@dbsrv1# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      4  98980 4558300 2306992    0    0     1    13    3    2  0  0 99  0  0

Answer 1

You need to do at least things to debug this:

1. Using a CLI ssh client rather than putty, run an 'ssh -v' to the server. See if you have any problems, and what errors the ssh session reports.

2. Using wireshark or tcpdump, get a packet capture of the session. You're probably getting a RST.

Per David's query, have you tried to access the server from another client? If not, perhaps the client is having a network issue.

If you're going through the firewall, the firewall will have its own session timeouts, as Kyle suggested above -- the 'keepalive' setting is a good workaround for that, if it is the issue. If the problem occurs with new sessions, a pcap will be the most expedient way of troubleshooting the problem.

Answer 2

Not sure I entirely understand:

"So, when I ssh(putty) to the box, if I haven't gotten on it in a while, putty immediatly throws a 'network connection reset' error."

So if you haven't been logged in, and then you start a new login, you get kicked out right away?

If that is not the case, and it just times out if you have been inactive. You could just set keepalive in putty.

You might also look in /var/log for the ssh logs and see if there is any information there.

Answer 3

This one was weird...

The webserver was a clone of another webserver from vmware server.

The original was shut down. Copied to new server. New machine brought up given new name given new IP Old machine brought back up.

THe problem with this scenario, the MAC address was the same on both, which resulted in very weird network behavior.