Server was compromised. Where to look?

Question

Recently we had a web server running Apache (2.2.3) compromised and are in the process of cleaning data and moving to another server to that we can rebuild the current one.

We want to try and figure out what was done after we've finished moving to a new server. We've got the easy stuff, the injected PHP code in numerous files and PHP inserted into the database.

At the same time we still have tags with-in the of html documents changed or a big blob of text inserted right after the opening tag in document. This happens on all HTML files sent to search engines be it from a static HTML file or generated from a PHP file. It will also redirect anyone coming from search engines to another site as long as their referer contains certain keywords.

If the page is cached in our reverse proxy this no longer happens, which has helped us narrow it down to something potentially in Apache. This is where I wonder "Where do I look?" we've looked for extra modules added in and checked in our config files and have yet to find anything that sticks out.

Where might be some other places to check so we can see what exactly is going on?

Answer 1

If you know the key words, you could grep all the files in your / root directory for them. Then manually go through the results.

Answer 2

Code inserted into files... the first thing I would think is compromised user. Simple FTP hack-job. In my experience (which is fairly considerable being a front-line sys admin for a web hosting company) weak password was it more times that people like to admit. There aren't a lot of ways to write to a file and this is one of them.

My earnest recommendation is to clean it all up and then:

1. Disable FTP altogether. Ideally through firewall settings.
2. Enforce strict passwords and also regular changing of passwords.
3. Often you will be tempted to allow loose permissions on files and directories. Never allow global write. There is always a better way.
4. Don't use PHP code that isn't from a well-respected source. Trash scripts from Google searches just aren't worth it.

If you really want to track down what event might have taken place.... search for IP addresses you aren't familiar with in your FTP logs. If you think it was a script hack, check your /tmp directory for hacker scripts. Any executable is bad news, as are any scripts at all. Check your applicable apache logs for references to the affected files, especially if those files are not often accessed.

Remember that you can wipe out code from files en masse with:

sed -ie s/regexorstring//g *

...run in the appropriate directory. That is just a find and replace (rudimentary) that will grab any string you want gone and will remove it. A little regular expressions in there will make it pretty easy.