8

So, we host a geoservice webserver thing at the office.

Someone apparently broke into this box (probably via ftp or ssh), and put some kind of irc-managed rootkit thing.

Now I'm trying to clean the whole thing up, I found the process pid who tries to connect via irc, but i can't figure out who's the invoking process (already looked with ps, pstree, lsof) The process is a perl script owned by www user, but ps aux |grep displays a fake file path on the last column.

Is there another way to trace that pid and catch the invoker?

Forgot to mention: the kernel is 2.6.23, which is exploitable to become root, but I can't touch this machine too much, so I can't upgrade the kernel

EDIT: lsof might help:

lsof -p 9481
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEss
perl 9481 www cwd DIR 8,2 608 2 /ss
perl 9481 www rtd DIR 8,2 608 2 /ss
perl 9481 www txt REG 8,2 1168928 38385 /usr/bin/perl5.8.8ss
perl 9481 www mem REG 8,2 135348 23286 /lib64/ld-2.5.soss
perl 9481 www mem REG 8,2 103711 23295 /lib64/libnsl-2.5.soss
perl 9481 www mem REG 8,2 19112 23292 /lib64/libdl-2.5.soss
perl 9481 www mem REG 8,2 586243 23293 /lib64/libm-2.5.soss
perl 9481 www mem REG 8,2 27041 23291 /lib64/libcrypt-2.5.soss
perl 9481 www mem REG 8,2 14262 23307 /lib64/libutil-2.5.soss
perl 9481 www mem REG 8,2 128642 23303 /lib64/libpthread-2.5.soss
perl 9481 www mem REG 8,2 1602809 23289 /lib64/libc-2.5.soss
perl 9481 www mem REG 8,2 19256 38662 /usr/lib64/perl5/5.8.8/x86_64-linux-threa d-multi/auto/IO/IO.soss
perl 9481 www mem REG 8,2 21328 38877 /usr/lib64/perl5/5.8.8/x86_64-linux-threa d-multi/auto/Socket/Socket.soss
perl 9481 www mem REG 8,2 52512 23298 /lib64/libnss_files-2.5.soss
perl 9481 www 0r FIFO 0,5 1068892 pipess
perl 9481 www 1w FIFO 0,5 1071920 pipess
perl 9481 www 2w FIFO 0,5 1068894 pipess
perl 9481 www 3u IPv4 130646198 TCP 192.168.90.7:60321->www.****.net:ircd (SYN_SENT)

2
  • 2
    Unless you upgrade the kernel, what's to stop the hacker repeating the hack as soon as you remove the rootkit? There may well be a trojan kernel module that hides processes.
    – pjc50
    Nov 12, 2010 at 11:06
  • this looks very similar to the irc ddos bot i just cleaned off my vps: serverfault.com/questions/639699/… Oct 26, 2014 at 12:01

5 Answers 5

38

If I can give you any advice, it is to stop wasting your time cleaning up. Make an image of the OS for forensic stuff for later, and just reinstall the server.

Sorry, but its the only secure way to resolving yourself from being rootkitted.

Later you can check the image, for certain reasons, why it happened.

From my own personal experience, I did this, and later found an internal user which had a SSH key containing the flaw of openssl in 2008.

I hope, it clears up the things.

Note:
If you are going to image/backup the server before reinstalling, be very careful, how you do this. As @dfranke said, boot from a trusted medium to backup.

You shouldn't connect to other machines from a rooted server, as great rootkits are known to be able to spread through trusted sessions such as SSH.

8
  • 11
    I strongly second this advice. You've found one rootkit, but you have absolutely no idea what else the attacker has tampered with. Once rooted, always rooted. Boot from a trusted medium, zero out the drive. Fdisk, format, re-install, doo dah, doo dah.
    – dfranke
    Nov 11, 2010 at 17:06
  • Yah... Good root kits run under your kernel.. No chance of deleting them, with out ALOT of effort
    – Arenstar
    Nov 11, 2010 at 17:47
  • 4
    +1 For any production system (really, any system) there is no reason to clean. Kill it with fire and rebuild.
    – phoebus
    Nov 11, 2010 at 18:03
  • 1
    +1 to reinstall. The rootkit probably messed with your binaries or kernel and everything you are seeing (fake paths, etc...) is probably smoke courtain from the rootkit to hide itself.
    – coredump
    Nov 18, 2010 at 0:25
  • 1
    Oblig. quote: "I say we take off and nuke the entire site from orbit. It's the only way to be sure." Nov 19, 2010 at 11:16
1

The commandline can be changed if the process alters argv[0]. Try ls -l /proc/[pid]/exe

From man 5 proc

this file is a symbolic link containing the actual pathname of the executed command. This symbolic link can be dereferenced normally; attempting to open it will open the executable. You can even type /proc/[number]/exe to run another copy of the same executable as is being run by process [number]. In a multithreaded process, the contents of this symbolic link are not available if the main thread has already terminated

ps auxwf | less gives you the "forest view" of processes which can show you what process launched this process (unless the rootkit is hiding it, or the app's parent has exited and it's been reparented to init).

This would be mostly academic and probably just a timewaster, but strings -n 10 /proc/[pid]/mem might be fun to watch scroll past. You could also echo 0x7 > /proc/[pid]/coredump_filter and use gdb gcore to force a coredump with everything possible in it, but then the process dies, which could block further analysis.

But definitely take Arenstar's advice. Back up data only, restore everything executable from backups, and start over. You should probably restore the website from backups as well, there could be malicious javascript added to every html or php file. If you're considering legal action, you'll want to just set the machine aside, unplug it from the net, and stop whatever it is you're doing until forensic experts have done their job.

2
  • Really great answer.
    – paul.ago
    Nov 12, 2010 at 9:21
  • unfortunately ls -l /proc/[pid]/exe returns the perl5.8.8 bin path, and ps/pstree says the process father is init, this thing looks really well hidden. I would definitely start over with a fresh install, but the original developer of the application running on it is out of the country for a while, so I was looking for a temporary solution. Really great answer by the way
    – paul.ago
    Nov 12, 2010 at 10:09
0

Try "cat /proc/[process id]/cmdline" Though if it is a true rootkit it may modify the kernel to hide itself better.

1
  • it gives me the same of ps aux "/usr/sbin/apache/logs" which is fake. Neither the /usr/sbin/apache directory exists
    – paul.ago
    Nov 11, 2010 at 16:39
0

You should reinstall, I agree. Have you tried escaping the characters in the path? Maybe one of those slashes is actually part of a filename and not a directory. At the very least you should use iptables to block outbound traffic to that host or IRC general until fixed. Check netstat as well.

1
  • nope, the path seems "real". Sadly iptables seems installed but the kernel module is missing, so I must recompile the kernel. I would probably go that way to fix the thing for a few days until I get in touch with the guy who has the app source code, so I can reinstall the server
    – paul.ago
    Nov 12, 2010 at 10:20
0

I'd think by now you have reinstalled. Your justing wasting time trying to track the processes and do forensics as chances of anything legally developing from it would be very minute and chances of finding the hacker would be futile anyway. Unless it just interests you to research and reverse rootkits which can be fun:)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .