4

I am running chkrootkit every night on 3:04. More than 1 year it was not returning any problems but 1 month ago this warning started to appear:

=======================
Checking `lkm'...                                           You have     1 process hidden for readdir command
You have     1 process hidden for ps command
chkproc: Warning: Possible LKM Trojan installed
=======================

Now I run chkrootkit every hour and at

3:02
3:03
3:04
3:05

No warnings at other times but 2-3 times per week chkrootkit returns this warning at 3:02 or at 3:04. The next scan after 1 minute doesn't retur this warning. Any ideas? Why this warning appears at 3 am? Is there any way how to run chkrootkit so that it shows more info about hidden process?

Yes, I know there are a lot of articles saying that it may be a false positive but why this warning started to appear so often?

Rkhunter, Clamav, LFD/CSF doesn't report any problems on the server.

0

I do not think it is possible for anyone here to properly figure out what is wrong with your system, unless we have direct access.

You can assume it is a false positive and that is what most likely is the case. In my experience I have had the odd false positive as well and after some investigation I had satisfactory confirmed to myself that indeed it was a false positive. Some processes can be hidden from the ps command for perfectly legitimate reasons. And are often short-lived.

However if you want to be sure you could do something as described below. I understand it is a bit extreme and may not apply at all in your situation, but I can't be sure of that, that is a decision you need to make.

I personally take a lot of wisdom on what to do in such and other situations, from Securing Debian Manual, especially in your case check Chapter 11 - After the compromise

The general advise is to unplug a compromised system from the network first and foremost, then back up the system. Then perform more investigation. This should be a separate activity and doesn't need to get in the way of restoring the system. It'd be a good idea to try and find out how it was compromised.

In the meantime completely wipe the system, re-install the OS from a non suspect read only source (say a dvd) and restore from backup only the data which you can be certain of is not infected. If such data is contained in repositories, then just retrieve it from there instead of the backup.

Assume any files on the system and its backups are suspect and try as much as possible to restore it without copying files from the backup. Restore configuration files by just repeating what you did, copy pasting text etc.

1

You have to look for the possible hidden script. The issue still exists, and most of the time, it's a false positive.

Tracking down the alert:

I've tracked the ps command before and after the script, in /etc/cron.daily/chkrootkit:

ls /proc/*/status -al > /tmp/lsproc.before.txt
ps axf > /tmp/psaxf.before.txt
ps auxwww > /tmp/psauxwww.before.txt
eval $CHKROOTKIT $RUN_DAILY_OPTS > $LOG_DIR/log.today.raw 2>&1
ls /proc/*/status -al > /tmp/lsproc.after.txt
ps axf > /tmp/psaxf.after.txt
ps auxwww > /tmp/psauxwww.after.txt

And the result is:

$ wc -l /tmp/*.txt
  218 lsproc.before.txt
  219 psauxwww.before.txt
  217 psaxf.before.txt

  218 lsproc.after.txt
  216 psauxwww.after.txt
  219 psaxf.after.txt

As you can see: the before is the same (ps aux has one first line for columns that ls /proc/* doesn't). So before, we have 218 processes. After, we have still 218 but only 215 (216 - 1) processes in ps. And I get the alert: You have 3 process hidden for ps command, so exactly the difference between /proc/ and ps.

What were those processes? I just diff the two psauxwww*:

my_user     16674  0.0  0.0  13976  1456 ?        S    05:25   0:00 sh /one/dedicated/script.sh
my_user     16690  0.0  0.0  13980  1392 ?        S    05:25   0:00 sh /one/dedicated/script.sh
my_user     16691  0.0  0.0  44284  2824 ?        S    05:25   0:00 ssh [...]

Not hidden, just missing!

So that's just one script, running every minute with my crontab, which is «hidden». But it's not hidden (as I found you can hide some processes): it just disappears during the chkproc test!

Solutions?

So far, I don't see any easy solution. chkrootkit is written in bash, but chkproc is compiled...

If you track the missing processes (most probably a cron as in my case), you may:

  • stop running the cron at the same time the chkrootkit does (not always possible, as sometimes you need to run your cron every minute, no matter what)

  • quit your script at the exact minute chkrootkit runs, hoping chkrootkit won't run more than one minute, for example in bash you'll add for chkrootkit running at 6:25:

    if [ "`date +%H%M`" = "0625" ]; then exit 0; fi

  • postpone the execution of chkrootkit a bit, by adding a sleep at the beginning of the /etc/cron.daily/chkrootkit script, for example if your other script takes less than 15 seconds:

    sleep 15

Not ideal, but if you need both crons to run at the same time, you know what to do.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.