Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

The Ubuntu Web server that I just inherited has been compromised. I'm a Windows admin and not all too familiar with Linux.

How would I go about finding out what keeps spawning perl processes? I killed a couple dozen instances of /usr/bin/perl but they are spawned again immediately. I've even renamed /usr/bin/perl but the processes still come back.

I recognize that I'll have to nuke the server, but I need to set up a replacement before I can take the website offline.

share|improve this question

4 Answers 4

check /etc/inittab for anything suspect:

0:2345:once:/usr/sbin/ttyload

would be one such script. You may have removed/renamed /usr/bin/perl, but, Debian/Ubuntu will also have a /usr/bin/perl5.10.1 (or whatever your current version you have installed). Perl4 may also still be on your achine.

share|improve this answer
    
I did check inittab. Nothing jumped out at me as suspect, but then again I wouldn't necessarily recognize something that is out of place. Based on a previous suggestion I checked out /proc for one of the perl processes, and the command line pointed to my /usr/bin/perl-remove (which didn't have the executable bits set). –  user46497 Jun 24 '10 at 13:21

Try chkrootkit, worked for me decades ago.
The other option was rkhunter.

These two can help you identify files added to you OS and delete, deactivate some of them. I used both to do some forensics about how the system was modified after a service was exploited.

HTH

share|improve this answer

What's the parent process ID of those new processes? Use ps -lp <PID>. The listing will have a column, PPID, which includes the process ID of the program that launched it.

Edit How about getting more information the hard way. Go to /proc//, and check the values of some of the soft links there. For example, "cwd" will point to the working directory of the script, and "exe" will point to the executable involved. That may give you some useful information. Alternately, attach strace to the process involved via strace -p <pid>, and see what calls it's making.


--Christopher Karel

share|improve this answer
    
PPID is 1 (init) –  user46497 Jun 22 '10 at 21:25
    
D'oh. Edited with some more in depth stuff. Try that. –  Christopher Karel Jun 22 '10 at 21:45

If you're sure it's hacked and this isn't a misbehaving or normal behaving for that matter service or cron job on the server then the first thing you need to do is get it off the net. Put a quick maintenance page or something notifying customers/users that the site is down for maintenance.

You aren't doing yourself any favors if your site gets tagged by Google, Microsoft, and AV companies as serving malicious content. Or if it's sending spam or being used as a stepping stone to attack other servers.

Make an image of the system and then nuke and rebuild to get it back on line as quick as possible. Now that the fire is out you can now focus on using that image and perhaps even restoring the image to a VM to do your forensics.

share|improve this answer
2  
+1 for getting this box off the net. If you're reasonably certain you've been compromised "down for maintenance" is almost certainly better than "up, harvesting your credit card numbers and sending you viagra spam"... –  voretaq7 Jun 22 '10 at 21:02
    
Thanks for the recommendation -- I've taken it down. I'd still like to find out what's going on before I put a potentially vulnerable web app back online. –  user46497 Jun 22 '10 at 21:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.