How to analyse logs after the site was hacked

Question

One of our web-projects was hacked. Malefactor changed some template files in project and 1 core file of the web-framework (it's one of the famous php-frameworks). We found all corrupted files by git and reverted them. So now I need to find the weak point.

With high probability we can say, that it's not the ftp or ssh password abduction. The support specialist of hosting provider (after logs analysis) said that it was the security hole in our code.

My questions:

1) What tools should I use, to review access and error logs of Apache? (Our server distro is Debian).

2) Can you write tips of suspicious lines detection in logs? Maybe tutorials or primers of some useful regexps or techniques?

3) How to separate "normal user behavior" from suspicious in logs.

4) Is there any way to preventing attacks in Apache?

Thanks for your help.

Answer 1

Like HopelessNOOb, I'd recomend getting some professional help with this.

When you know that a system has been compromised you can't rely on any data stored on it. Further, if the compromise was via HTTP, then there's probably not enough information in the standard logs to be able to isolate what hapenned. This is why people who are serious about webserver security will use something like mod_security to get much more detailed logs, and run a host based IDS to detect compromises.

The support specialist of hosting provider (after logs analysis) said that it was the security hole in our code

If he/she can make that claim then he/she must already know the answer to your question - either that or the claim is not substantiated and they really mean to say that they couldn't find anything wrong with their stuff.

Your objective should be to get your site back online and secure. To make a site secure, you need to plug every hole - but to compromise a site you only need to find one hole: you know you have at least one hole in your site, but you need to fix any which exist - which means you need to take a very different approach to your site security. Even of you can identify and fix the vulnerability which was exploited this time, the evidence shows that you don't have the processes and skills in place to have any sort of confidence that this was an isolated incident.

Answer 2

1) You can have a look at these links for some software to help you, though I've not used most of them myself. But given that you've got timestamps on the files that were altered, start by looking at the log files for those dates and time. Grep for the filename of one of the files altered; that should give you the ip address of the malefactor, and then you can start grepping for that.

2 and 3) The first thing you need to do is to find out what is normal for your own site. One way to do that is to have a standard analyzing tool, such as awstat, go through your logs daily. Your main problem now appears to be that you don't know what is normal for your site, so you have no way of knowing what is abnormal.

4) There are ways of preventing some attacks, though of course it's an arms race. But just by removing the most common exploits you'll make it far less likely that you'll be the target of a random attack. You should make sure that your installation has the latest security patches for your OS, for apache and for php. You should also have a look at mod_security, which allows you to both log suspicious activity and to e.g. require that a client send headers such as User-Agent, which many crack tools don't.

Also, I'd advise you to consider sending your logs, especially error logs and any mod_security logs, to another server if possible, so that even if somebody manages to get into your server they won't be able to edit the log files to hide their tracks.

Answer 3

You got a good answer from JennyD, but I'd suggest a different approach.

Hire an expert.

There are security experts who actually do this for a living and will do a far better, far quicker job of determining how you were hacked and where you're still vulnerable. Check out the local security market, and I'd be surprised if you can't find a decent price on a consulting services to analyze the logs for you, as well as run at least a basic penetration and vulnerability scan on your system(s). You'll get a much more complete picture of what you need to do to secure yourself, and you'll be able to get back to spending your time within your areas of expertise.

Answer 4

In addition to the guidance here, these topics are discussed in depth over on the security stack exchange. Have a read of the question on Apache hardening: https://security.stackexchange.com/questions/77/485

Answer 5

You might want to check out LORG - https://github.com/jensvoid/lorg - a PHP-CLI tool (GPL) designed for post-attack forenscis of Apache access_logs. It uses several methods (signature-based, statistics, learning-based) to detect attacks against web applications in HTTPD log files.

It tries to group those suspicious activities into sessions and applies robot detection techniques (typically, an attacker might run an automated scan and then come back later to manually check what it has found). Also, it uses geomapping and DNSBL-lookups to see if the attacks orignate from a certain country or botnet.

Afterwards you can choose various mechanisms to determine if an attack was successful: active replays and matching for signatures, HTTP response codes or outlyingness in the size of responses (the bytes-sent field).

Obviously, if the log files have been altered or deleted, things might get difficult (LORG has the option to perform a simple tamper check to find extraordinary time windows with no activity, but this won't really help you).

As for preventing attacks, i'd go with mod_security (if yourhosting provider allows it).

Answer 6

1) What tools should I use, to review access and error logs of Apache? (Our server distro is Debian).

Check out the man page for syslog and some of the tools in the links below.

• syslog
• multitail
• log analysis
• open-source log monitoring

2) Can you write tips of suspicious lines detection in logs? Maybe tutorials or primers of some useful regexps or techniques?

This is a bit more complicated and would be a huge topic all on its own but to get started I would look into what the tools above do for log analysis and what they are looking for. The link below should point you in the general direction to start learning.

• snort-linux

3) How to separate "normal user behavior" from suspicious in logs.

Practice, practice, practice. Also, learn a set of tools well and understand what is happening "under the hood"

4) Is there any way to preventing attacks in Apache?

No. The world we live in is chaotic and driven. As long as people constantly want new features for less money there will be corners cut and buffers overflown. With this said the Apache software itself is probably fairly safe (highly audited) but you are at the mercy of whichever libraries you choose to use and permissions you may accidentally or unknowingly set.

Finally, If you have the money and you are using your site for something non-trivial or rely on the revenue created from the site the best course is to hire an expert as mentioned in other comments.

Aside: I am very disappointed in most of the answers which point the OP to hiring an expert. While this may be sound advice to preface an answer with it does not answer any of the OP's carefully laid out questions.