Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

all. I have these weird servers can not explained as follow:

htop

1  [|||||||||||||||                              28.5%]     Tasks: 53 total, 1 running
2  [||||||||||||||||                             31.1%]     Load average: 0.00 0.00 0.00 
3  [||||||||||||||||                             30.5%]     Uptime: 211 days(!), 02:21:04
4  [|                                             0.7%]
Mem[||||                                   171/16077MB]
Swp[                                         0/11610MB]

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command                                                   
    1 root      20   0  8352   840   704 S  0.0  0.0  1:02.48 init [2]
23764 root      20   0 10584  1364  1172 S  0.0  0.0  0:00.00  `- bash -c while sleep 0.000001; do echo 29150 | md5sum ; done

top

top - 01:36:46 up 211 days,  2:40,  5 users,  load average: 0.00, 0.00, 0.00
Tasks: 108 total,   2 running, 106 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.8%us, 18.0%sy,  0.0%ni, 77.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16463184k total,   797364k used, 15665820k free,   122992k buffers
Swap: 11889656k total,        0k used, 11889656k free,   499496k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                   
1 root      20   0  8352  840  704 S    0  0.0   1:02.48 init                                                       
2 root      20   0     0    0    0 S    0  0.0   0:00.15 kthreadd                                                   
3 root      RT   0     0    0    0 S    0  0.0   0:00.14 migration/0                                                
4 root      20   0     0    0    0 S    0  0.0   0:00.22 ksoftirqd/0                                                
5 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0                                                 
6 root      RT   0     0    0    0 S    0  0.0   0:00.16 migration/1                                                
7 root      20   0     0    0    0 S    0  0.0   0:00.09 ksoftirqd/1                                                
8 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/1                                                 

the server acted that status some days before, so I try to use the "while...md5sum" offer the cpu pressure, but not only that while's cpu/mem usage were 0%, but actually all the others' usage were 0%.

obviously when I killed that while loop, the htop bar went down to true 0% (the server don't really have much work to do).

and I double check the "md5sum which htop" (and top) on the other NORMAL server, they have the exactly binary/md5 result.

so, any idea ? Am I so deadly rootkitted? I have used rkhunter / chkrootkit already, no clue.

share|improve this question
    
did you consider a reboot? –  the-wabbit Nov 19 '11 at 22:34
    
good point view. actually, I have a group of these servers. some act that weird, the others seemed all right. I did reboot one of the weird s, turned out, I ran the while loop on it, the top/htop functioned as normal... but I keept this weird one not rebooting. Then I came to post this question. I am so doubt of being rooted, that is so unacceptable for these server/service . –  kiiwii Nov 20 '11 at 1:17

3 Answers 3

To check if top or associated libraries are hiding processes due to a rootkit, you can compile a static version of top on another system. Then copy that version over and run it. If you've been root kitted, the hidden processes should show up in that static top since it won't be using any of the rootkit libraries.

Some suggestions in regards to determining what else might be causing the problem:

  • Disable as many unnecessary services (networking, iptables, auditd, selinux, sendmail, nfs, netfs, nscd, etc.) as possible to limit where the extra CPU cycles could be being used.
  • Look in /var/log/* to see if anything is spitting out errors.
  • Enable logging or more verbosity in your services
  • Use a program like dtrace or systemtap to see what is going on in
share|improve this answer
    
thank a lot for the information. follow your instruction, I found this static-linked version of htop(htop.sourceforge.net/htop-0.6.6-static.gz), and ran on the weird box. over ran the CPU with that loop, but still, display all process USING 0% CPU. what a awful day and night I had... can we assume the debian/linux issue ? all my boxes running debian squeeze: "Linux XXX 2.6.32-5-amd64 #1 SMP Mon Mar 7 21:35:22 UTC 2011 x86_64 GNU/Linux" rebboot did helped. but I cannot accept that windows moto "restart/reboot to rescue the world"... so, at the end, all I have to do is a fully reinstall? –  kiiwii Nov 20 '11 at 8:55
    
Any errors in /var/log/messages? It could be your HD is having IO problems and causing system calls to stall waiting on IO. Have you tried enabling smartd and running smartctl --all /dev/sda or whatever your hard drive is? Yes, I think you can eliminate rootkit as the cause. –  Andrew Case Nov 21 '11 at 9:01
    
dropbear accept my user/pass, but failed to provide the shell. even the shell I left inside the screen of that box became unfunctional. simple ls OR w command failed to execute. no other way but only snmpd to my zenoss-server, I could still know about my server's status was the extremely high cpu load / usage. no extra network, or IO or memory. just cpu. after one hour or later, the load went down, and let me back to the box, check all the box status. nothing special, like nothing happened, no proces causing no problem like ghost never here before .... what a horrible box ... –  kiiwii Nov 21 '11 at 11:10
    
I reported this issue to the company, we decided to do a fully reinstall. thanks all . –  kiiwii Nov 21 '11 at 11:11
    
SORRRY, not "much", but "NOT much",my apology. I might want to say "little" OR "not much". definitely NOT "much", all. sorry again. I hope no rootkit, too. no clues or traces from /var/log/auth.log||syslog||OTHTER.LOGS. also, I have no unnecessary services on that box, and all the servers are doing their own job and under NOT very much pressure/load.So you may wonder why I am curious about this weird event, it is only skin-deep, but for more information, the box's load sometimes turned from around 0-1 to 40-100, and sshd rejected my login(port 22 close instantly), –  kiiwii Nov 21 '11 at 11:15

Look at this: 18.0%sy, this is time your server spent executing kernel calls. This parameter is unusually high on your server. This time can be not directly related to userspace processes so top shows no process consuming the CPU. The reason can be for example a big number of similar iptables rules on your system. If you try to block a botnet with 20+K IP addresses using plain iptables rules you will get system time parameter in top (and actual CPU usage in kernel mode) that high.

share|improve this answer
    
thanks for your pointing out. but when I killed that while loop, cpu usage went down instantly. And I do have the iptables in white-lists mode. but " 176 8328 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 " It is very little, though. –  kiiwii Nov 19 '11 at 20:17
    
and, of course, when I run that while loop on other normal servers have the same hareware/debian. 29761 root 20 0 10584 1364 1172 S 18 0.0 0:06.70 bash much unlike the weird one, it jus show no process using no cpu(ALL 0%), not the system process, not bash... but cpu actually was pretty much used. –  kiiwii Nov 19 '11 at 20:38

On what tty are you running the while loop? I have a vserver with 8 CPUs, I'm logging in via ssh, the loop is running at 4 CPUs at first, runs only on 2 CPUs later but the load goes up to 1.5 while %CPU remains at 0. Can you check /proc/loadavg (if you have this) while the loop is running.

share|improve this answer
    
ssh THATSERVER "while...md5..." –  kiiwii Nov 19 '11 at 20:16
    
I can reduce the load from 1.5 to 0.5 when I redirect the output to /dev/null. What does your /proc/loadavg report? –  ott-- Nov 19 '11 at 20:33
    
all around 0.13 0.03 0.01, bro. /proc/loadavg accommodated of top/htop and uptime and on and on ... –  kiiwii Nov 19 '11 at 21:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.