My server was under heavy load around 400 and above. Here is the link of Server Fault question
I was able to see the rm command and xargs running in top output, in which I was the one and only user logged in.
I tried to Kill that process but that didn't work.
I tried to write an iptables rule to make a default policy to drop and allow only my IP to communicate, but before doing that iptables went missing. I installed it again but it showed:
FATAL: Could not load /lib/modules/2.6.32-5-vserver-amd64/modules.dep: No such file or directory iptables v1.4.14: can't initialize iptables table `filter': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
When I tried to shutdown the server, I was getting time out messages. Rebooting also didn't work.
Once load came down I performed a chrootkit scan and here is the result. It shows many missing modules and hidden files.
Searching for suspicious files and dirs, it may take a while... The following suspicious files and directories were found: /usr/lib/pymodules/python2.6/.path /usr/lib/pymodules/python2.7/.path /usr/lib/node_modules/npm/.npmignore /usr/lib/node_modules/npm/.travis.yml /usr/lib/node_modules/npm/node_modules/fast-list/.npmignore /usr/lib/node_modules/npm/node_modules/fast-list/.travis.yml /usr/lib/node_modules/npm/node_modules/fstream/.npmignore /usr/lib/node_modules/npm/node_modules/fstream/.travis.yml /usr/lib/node_modules/npm/node_modules/graceful-fs/.npmignore /usr/lib/node_modules/npm/node_modules/lru-cache/.npmignore /usr/lib/node_modules/npm/node_modules/minimatch/.travis.yml /usr/lib/node_modules/npm/node_modules/node-uuid/.npmignore /usr/lib/node_modules/npm/node_modules/nopt/.npmignore /usr/lib/node_modules/npm/node_modules/slide/.npmignore /usr/lib/node_modules/npm/node_modules/tar/.npmignore /usr/lib/node_modules/npm/node_modules/tar/.travis.yml /usr/lib/node_modules/npm/node_modules/.bin /usr/lib/node_modules/npm/test/packages/npm-test-files/.npmignore /usr/lib/node_modules/npm/test/packages/npm-test-ignore/.npmignore /usr/lib/node_modules/npm/node_modules/.bin
Checking `lkm'... You have 3086 process hidden for readdir command SIGINVISIBLE Adore found chkproc: Warning: Possible LKM Trojan installed
Do I need to investigate further to confirm that it was an attack?
How can I get details about the way attacker got in?