Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a linux server that has started acting a little funny...

First of all the box is way under utilized. It's a database server, with a very low load. I'll put vmstat below and top...

  1. I'm running centos 5.4
  2. This server has been running fine for weeks
  3. It's firewalled, it's unlikely it's been hacked (chkrootkit agrees).
  4. Load is really low.
  5. I'm on a Gigabit switch with the server.
  6. I'm not having this problem with any other server.
  7. I have remote syslog set up on another box and it's not indicating any corresponding errors.

So, when I ssh(putty) to the box, if I haven't gotten on it in a while, putty immediatly throws a 'network connection reset' error.

Randomly it just kicks me out (or network drops)

a steady stream of pings shows now interruption.

--- vdbsrv1 ping statistics ---
80620 packets transmitted, 80619 received, 0% packet loss, time 23838ms
rtt min/avg/max/mdev = 0.234/0.273/1.741/0.043 ms, ipg/ewma 0.295/0.297 ms

SQL queries ("Select 1") to the box take about 5 seconds, even though SQL is reporting something like a .04 second execution time, so the other 4.96 seconds is a mystery

Anyone have any ideas?

top - 11:49:28 up 12 days, 17:05,  4 users,  load average: 0.00, 0.03, 0.01
Tasks: 150 total,   1 running, 149 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 99.3%id,  0.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8174028k total,  8074924k used,    99104k free,  4558300k buffers
Swap: 19464184k total,        4k used, 19464180k free,  2306952k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    1 root      15   0 10348  692  576 S  0.0  0.0   0:02.13 init

root@dbsrv1# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      4  98980 4558300 2306992    0    0     1    13    3    2  0  0 99  0  0
share|improve this question
    
Does this happen when logging in from other computers/location? –  David Jan 27 '10 at 18:17
    
When you say queries take 5s, is it measured as time psql -c query or equivalent, or does it take the network into account? –  Tobu Jan 27 '10 at 18:44
add comment

3 Answers

up vote 0 down vote accepted

You need to do at least things to debug this:

  1. Using a CLI ssh client rather than putty, run an 'ssh -v' to the server. See if you have any problems, and what errors the ssh session reports.

  2. Using wireshark or tcpdump, get a packet capture of the session. You're probably getting a RST.

Per David's query, have you tried to access the server from another client? If not, perhaps the client is having a network issue.

If you're going through the firewall, the firewall will have its own session timeouts, as Kyle suggested above -- the 'keepalive' setting is a good workaround for that, if it is the issue. If the problem occurs with new sessions, a pcap will be the most expedient way of troubleshooting the problem.

share|improve this answer
    
This was helpful. Turned out since it was a cloned virtual machine, both the clone and the original were using the same MAC. Not a good thing. –  Benny B Nov 30 '10 at 19:00
add comment

Not sure I entirely understand:

"So, when I ssh(putty) to the box, if I haven't gotten on it in a while, putty immediatly throws a 'network connection reset' error."

So if you haven't been logged in, and then you start a new login, you get kicked out right away?

If that is not the case, and it just times out if you have been inactive. You could just set keepalive in putty.

You might also look in /var/log for the ssh logs and see if there is any information there.

share|improve this answer
    
I agree this sounds like a timeout issue. Another way to test this is to just run something like top that will constantly have traffic over the session. If you can leave this running for a while and see if that keeps the connection alive. –  3dinfluence Jan 27 '10 at 18:48
    
same here, looks like timeout. useful line to generate some traffic: while true; do echo .; sleep 10; done –  rytis Jan 27 '10 at 19:02
add comment

This one was weird...

The webserver was a clone of another webserver from vmware server.

The original was shut down. Copied to new server. New machine brought up given new name given new IP Old machine brought back up.

THe problem with this scenario, the MAC address was the same on both, which resulted in very weird network behavior.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.