How can I figure out which site on my server is getting swamped with traffic?

Question

I'm running a linux server with about 60 websites on it that is crashing under heavy load. Is there an easy way to see which Apache virtual host is getting the most traffic?

Answer 1

Make sure you have mod_status.so loaded within your apache modules then look for/add the above to your httpd.conf:

# Uncomment the following lines to enable mod_status support:
#
ExtendedStatus On

<Location /server-status>
SetHandler server-status

Order Deny,Allow
Deny from all
Allow from YOUR_IP_HERE
</Location>

This will allow you to see all the pages being used load domain within your http server.

To access it use http://your_ip/server-status and only the ip defined at Allow from YOUR_IP_HERE will be able to view it.

Aside from that, like recommended i would use netstat, the server logs and mod_backdoor (serves to get information from an apache that's too sick to respond normally).

Taken from the mod_backdoor.txt

To compile/install mod_backdoor, perform the following operations:

# apxs -c mod_backdoor.c
# apxs -i mod_backdoor.la

To enable mod_backdoor, add something like the following to your conf file:

loadmodule backdoor_module modules/mod_backdoor.so
<IfModule mod_backdoor.c>
    BackdoorAddress 127.0.0.1:65535
</IfModule>

Although the controls below are redundant with the BackdoorAddress shown above, it may be useful as an example. You could specify 0.0.0.0:port for BackdoorAddress then use mod_access directives to control which clients can use the back door.

<VirtualHost 127.0.0.1:65535>
    <location />
        order deny,allow
        allow from 127.0.0.1
        deny from all
    </location>
</VirtualHost>

Answer 2

I've had some good results with wtop / logrep in the past for a box with a single site on it. I don't see any reason why it wouldn't scale to multiple sites, and has support for a url field that could be filtered on.

Answer 3

Depends on your setup. Does each VirtualHost have its own IP? I would think, probably not, so I'd check into netstat processes to see if there is one IP that it hitting your server that is causing a DDOS of some sort:

netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

Answer 4

Parse the apache access log with something like: http://awstats.sourceforge.net/

Or use ngrep:

sudo ngrep -qp 'GET|POST' port 80

Answer 5

We recently had our web server being crippled by 220% mysqld CPU spikes. We used mytop to match up the 'spikes' in CPU with the queries being run and narrowed down the culprit site.

If mysql is involved then check out the slow query log as well, might help you find the slow queries and then you can use "EXPLAIN" to help you optimise them.

Answer 6

To add to MikeR's answer recommending wtop / logrep, you could get requests per domain (15 most requested domains):

logrep -o 'count(*),domain' -s 15:1:d access.log

You can also use logrep in tail mode, which can be great for simplifying the data and making patterns easier to spot:

logrep -m tail -o 'msec,method,status,domain,refdom,uas' access.log

If the above looks potentially helpful, then also checkout "classes".