Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So I got hit by a script kitte... Fortunately the box is Ubuntu and was able to replace w/ binaries from a comparable system, however,

Some of the files I couldn't delete, and am still stumped on this. The hijacked files are sitting in the /_bin directory which is writeable by root.

nathan@db-0:~$ ls -ld !$
ls -ld /_bin
drwxr-xr-x 2 root root 4096 Mar 12 18:00 /_bin

Ok, those are the perms on the directory, now for the files within:

nathan@db-0:~$ ls -l /_bin
total 268
-rwxr-xr-x 1 root root  39696 Nov 19 22:25 ls
-rwxr-xr-x 1 root root 119800 Mar 31  2012 netstat
-rwxr-xr-x 1 root root 101240 Dec 12  2011 ps

Now when I try to delete one of these files (as root):

root@db-0:/home/nathan# rm /_bin/ls
rm: cannot remove `/_bin/ls': Operation not permitted

Or if I try to delete the entire _bin directory (again as root):

root@db-0:/home/nathan# rm -rf /_bin
rm: cannot remove `/_bin/ls': Operation not permitted
rm: cannot remove `/_bin/netstat': Operation not permitted
rm: cannot remove `/_bin/ps': Operation not permitted

So how can I delete these files?

Edit:

Sure enough the immutable bit has been set, however, removing it does not let me delete the files.

root@db-0:/home/nathan# lsattr /_bin
s---ia--------- /_bin/ls
s---ia--------- /_bin/netstat
s---ia--------- /_bin/ps

root@db-0:/home/nathan# chattr -R -i /_bin
root@db-0:/home/nathan# lsattr /_bin
s----a--------- /_bin/ls
s----a--------- /_bin/netstat
s----a--------- /_bin/ps

root@db-0:/home/nathan# rm -rf /_bin
rm: cannot remove `/_bin/ls': Operation not permitted
rm: cannot remove `/_bin/netstat': Operation not permitted
rm: cannot remove `/_bin/ps': Operation not permitted

Also verified /_bin doesn't have immutable bit:

root@db-0:/home/nathan# lsattr -d /_bin
--------------- /_bin
share|improve this question
6  
Reinstall the system from scratch, don't try to repair it. You'll be never sure to have eliminated everything. –  SvW Mar 12 '13 at 18:16
    
Can you do strace -v rm /_bin/ls and as it's longer output, put it on the web like pastebin.com? –  ott-- Mar 12 '13 at 18:34
2  
I say we take off and nuke the entire site from orbit. It's the only way to be sure. youtube.com/watch?v=aCbfMkh940Q –  Deer Hunter Mar 12 '13 at 19:05
    
The answer you accepted says there are other extended attributes that might prevent removal... there might also be /_bin/.* files with funky attributes. –  vonbrand Mar 12 '13 at 22:14
    
Good man @vonbrand, though that wasn't the case here an old colleague once told me of a case where there was a ... directory - tricky kitty! –  quickshiftin Mar 13 '13 at 0:06

2 Answers 2

up vote 7 down vote accepted

Most likely the attacker has set the immutable attribute on the files and directory. This is commonly done by rootkits to make cleanup more difficult.

To confirm this, try:

lsattr /_bin

To remove the immutable attribute, use:

chattr -R -i /_bin

You'll also want to clear the a and s attributes, since these may affect your ability to remove the files.

chattr -R -i -a -s /_bin

See the chattr man page for a full explanation of what all the attributes are and what they do.

share|improve this answer
    
Hi Michael, thanks for this! I'm still having trouble though, can you scope out the edit above? –  quickshiftin Mar 12 '13 at 18:24
    
Looks like you should clear out the a and s attributes too. –  Michael Hampton Mar 12 '13 at 18:26

Looks like the sticky bit is still there.

chmod -t /_bin
share|improve this answer
    
That is irrelevant... –  vonbrand Mar 12 '13 at 22:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.