My server was under heavy load around 400 and above. Here is the link of Server Fault question
I was able to see the rm command and xargs running in top output, in which I was the one and only user logged in.
I tried to Kill that process but that didn't work.
I tried to write an iptables rule to make a default policy to drop and allow only my IP to communicate, but before doing that iptables went missing. I installed it again but it showed:
FATAL: Could not load /lib/modules/2.6.32-5-vserver-amd64/modules.dep: No such file or directory iptables v1.4.14: can't initialize iptables table `filter': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
When I tried to shutdown the server, I was getting time out messages. Rebooting also didn't work.
Once load came down I performed a chrootkit scan and here is the result. It shows many missing modules and hidden files.
Searching for suspicious files and dirs, it may take a while... The
following suspicious files and directories were found:
/usr/lib/pymodules/python2.6/.path /usr/lib/pymodules/python2.7/.path
/usr/lib/node_modules/npm/.npmignore
/usr/lib/node_modules/npm/.travis.yml
/usr/lib/node_modules/npm/node_modules/fast-list/.npmignore
/usr/lib/node_modules/npm/node_modules/fast-list/.travis.yml
/usr/lib/node_modules/npm/node_modules/fstream/.npmignore
/usr/lib/node_modules/npm/node_modules/fstream/.travis.yml
/usr/lib/node_modules/npm/node_modules/graceful-fs/.npmignore
/usr/lib/node_modules/npm/node_modules/lru-cache/.npmignore
/usr/lib/node_modules/npm/node_modules/minimatch/.travis.yml
/usr/lib/node_modules/npm/node_modules/node-uuid/.npmignore
/usr/lib/node_modules/npm/node_modules/nopt/.npmignore
/usr/lib/node_modules/npm/node_modules/slide/.npmignore
/usr/lib/node_modules/npm/node_modules/tar/.npmignore
/usr/lib/node_modules/npm/node_modules/tar/.travis.yml
/usr/lib/node_modules/npm/node_modules/.bin
/usr/lib/node_modules/npm/test/packages/npm-test-files/.npmignore
/usr/lib/node_modules/npm/test/packages/npm-test-ignore/.npmignore
/usr/lib/node_modules/npm/node_modules/.bin
Checking `lkm'... You have 3086 process hidden for readdir command SIGINVISIBLE Adore found chkproc: Warning: Possible LKM Trojan installed
Do I need to investigate further to confirm that it was an attack?
How can I get details about the way attacker got in?
ps aux | awk '$8 ~ /D/ { print $0 }'
?