Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a RHEL server. Today it suddenly became slow. Symptoms - It was responding slow to ping queries from other server. When I try to login using ssh, it was taking about 10 seconds to login. I was able to resolve the problem by doing some guess work. I killed one process which I thought was culprit. Which resolved the problem. Though I would like to know what's proper approach to detect the culprit in such kind of "slow server" situations. Le me know proper way to resolving such slowness issues and decting the process causing the slowness.

These were the conditions when the server was slow -

# vmstat 3 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  1    176 6730868 285052 4899676    0    0     3     4    0    0  1  1 97  1  0
 0  0    176 6751576 285064 4899704    0    0     0   115 15307 37171  1  1 96  3  0
 0  0    176 6751948 285068 4899700    0    0     0    23 14813 39559  1  1 98  1  0

# top

top - 16:38:18 up 150 days, 19:36, 64 users,  load average: 1.68, 1.46, 1.44
Tasks: 1287 total,   2 running, 1284 sleeping,   1 stopped,   0 zombie
Cpu(s):  1.3%us,  1.7%sy,  0.1%ni, 95.9%id,  0.7%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  16620824k total,  9867124k used,  6753700k free,   287424k buffers
Swap:  8193140k total,      176k used,  8192964k free,  4898996k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26258 khk       34  19  130m  47m 7088 S 11.2  0.3 385:32.42 edm

Though I would like to know what's proper approach to detect the culprit in such kind of "slow server" situations. Le me know proper way to resolving such slowness issues and decting the process causing the slowness.

Note - Since ping response was slow, we first suspected network issues. But it was not network issue. We confirmed that by switching the network cable with adjacent server which was also on same subnet.

share|improve this question
    
Try using iotop and atop? These measure the disk usage. You'll also want to check the kernel time. Try using htop and seeing how much red is on the CPU's if it's a lot use perf top to troubleshoot what is causing high kernel time. –  josten Oct 29 '13 at 22:45
2  
To be honest, your machine looks perfectly fine. A load of 1.68 is no problem, your CPU is 95% idle (which is somewhat contradictionary, but the 0.7% wait is fine too). Are you sure it wasn't a network problem? –  JvO Oct 30 '13 at 0:23
    
Seconding atop, it can replace usage of top, iotop, vmstat, etc. For an introduction see lwn.net/Articles/387202 –  sciurus Oct 30 '13 at 0:30
    
There is always a chance that something could trigger a bug on the NIC driver. If that is the case, you should be able to see something standing out in your kernel messages. –  jijix Oct 30 '13 at 16:36
    
We know which process is causing this slowness. –  user1149518 Oct 30 '13 at 21:59
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.