Calculate request frequency and bandwidth usage for specific resources (CDN related)

Question

I am currently looking into the viability of using a CDN on my site (~2mill page views per month). However, before looking at the myriad packages available it makes sense to find out exactly what I need rather than stabbing in the dark with an 'finger in the air' estimate.

So, does anybody know of any utilities that will be able to extract out the volume of requests etc to specific resources (i.e. *.css, *.js, *.gif etc etc) from my Webserver (Apache) logs (or maybe somewhere else).

With this information I will know how many requests per second the CDN will have to support, and more importantly how much bandwidth I will need - From this I should be able to choose the correct CDN package for my needs.

Thanks in advance

Answer 1

At the risk of going all UNIX-y,

egrep '\.css|\.gif|\.js' httpd.access | awk '{print $10}'| perl -n -e '$sum += $_ ; print "$sum\n" ;'| tail -1

will add up all the bytes delivered in .css, .gif and .js requests, and print the total. If this doesn't work for you, try

egrep '\.css|\.gif|\.js' httpd.access | head

to make sure it's printing just the relevant lines, then

egrep '\.css|\.gif|\.js' httpd.access | awk '{print $10}' | head

to make sure it's printing only the bytes counts of the relevant lines (should be the field after the HTTP return code), because like most UNIX text hacks, this one depends quite strongly on the format of the file we're operating on. One you have the byte count, the bandwidth is the byte count divided by the time between the first and last lines in httpd.access.