Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I just noticed a weird directories stored on both in /tmp within two of my CentOS boxes.

On one machine the temp directory is called /tmp/www4-679109 while on my second machine the temp directory is /tmp/sos_e6X9_3.

Both of these suspicious directories contain four subdirectories: etc proc sys var .

The etc folder on this suspicious directory tree contains an identical copy of my master submit.cf sendmail configuration file. /tmp/sos_e6X9_3/etc/mail/submit.cf and /etc/mail/submit.cf. This makes me think that some how sendmail was used to relay mail. Though I can't verify the maillog files were inconclusive is this was the case.

Also this suspecious wierd directory contains some sort of audit log files within it. (/tmp/sos_e6X9_3/var/log/audit/audit.log.1)

Snippet of contents of a log file:

type=LOGIN msg=audit(1293719401.416:2772543): login pid=6662 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=557197
type=LOGIN msg=audit(1293719401.417:2772544): login pid=6660 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=557198
type=LOGIN msg=audit(1293719401.418:2772545): login pid=6649 uid=0 old auid=4294967295 new auid=599 old ses=4294967295 new ses=557199
type=LOGIN msg=audit(1293719401.420:2772546): login pid=6665 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=557200
type=USER_ACCT msg=audit(1293719401.423:2772547): user pid=6668 uid=0 auid=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='PAM: accounting acct="root" : exe="/usr/sbin/crond" (hostname=?, addr=?, terminal=cron res=success)'
type=USER_START msg=audit(1293719401.424:2772548): user pid=6653 uid=0 auid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='PAM: session open acct="root" : exe="/usr/sbin/crond" (hostname=?, addr=?, terminal=cron res=success)'

What is concerning me more with this suspicious files and directories on my CentOS boxes is that it appears that their is a procfs like directory in /tmp/sos_e6X9_3/proc. The only difference with /tmp/sos_e6X9_3/proc and /proc is that in the /tmp/sos_e6X9_3/proc there aren't any viewable process information.

I have kept both CentOS machines up to date and did not make any change to sendmail that would somewhow make it vulnerable, (unless sendmail is vulnerable right off the box).

Does anyone have have any ideas/feedback on why these suspicious temp files and folders were created?

UPDATE: It appears that the suspicious files were generated by the sosreport utility when we were working with Red Hat support on a different issue. I checked the log and the timestamps match with the timestamp with the suspicious files and directories. Thanks for your support and feedback. You guys rock!

share|improve this question
    
That does look suspicious. The uid 4294967295 is nfsnobody and they should not be logging in. What user has the uid 599? –  Mark Wagner Apr 18 '11 at 19:16
    
The user that has the uid 599 is an account that I created so I can run a dedicated Apache instance to run under this acccount. Though when this account was created, it was assigned a random password and the shell was changed to /sbin/nologin. sitenameapache:x:599:774::/www/sitenameapache:/sbin/nologin –  Alpha01 Apr 18 '11 at 20:58

3 Answers 3

up vote 1 down vote accepted

The sos directory you found in /tmp is created when the command sosreport is executed. This is a Red Hat utility that collects system information that is useful for Red Hat Support when troubleshooting your system. This, as far as I know, isn't useful on CentOS but since CentOS is based off of RHEL that's why it's there (assumption).

Edit: I missed your bold print stating you found out it deals with sosreport.

share|improve this answer

I would definitely run 'chkrootkit' and 'rkhunter' immediately. If you can download a binary for netstat, lsof, ls, less and ps and run them directly then you'll have a better idea what the boxes are doing.

Note that these downloaded, clean binaries should only be trusted for that login session. Download them each time you need them. I've seen files overwritten on each login after a compromise.

If you can firewall everything except your admin IP for SSH for a few minutes then all the better while you run these tests.

During your careful search around the machine look for ANYTHING different. From the formatted colour output of 'top' to the speed 'w' loads up.

share|improve this answer

No matter whether you actually find a rootkit, I would:

  1. change all relevant passwords in the environment,
  2. reboot the boxen on a live CD and dd the full disk image across the network to somewhere for later analysis,
  3. scrub the disks using dd if=/dev/zero of=/dev/<whatever>. Make sure you overwrite the MBR,
  4. reinstall using a public repository,
  5. analyze the image to try to find out what happened, and
  6. report the conclusions to some relevant clearinghouse.

Yes, this is a bit paranoid, but is typically a faster route to revovery than trying to study the problem and be smart.

It seems from OP that this is a mail server. If so, you have an additional problem in that the box may be sending out contaminated mail. I would not release my mail queues until I was absolutely sure it was not being rewritten enroute.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.