3

We have a busy webserver (> 5 million hits/day) that serves about 500,000 unique files. We're running Apache on FreeBSD 7.2. According to iostat -x, the bottleneck seems to be the seek speed of the drives (we're running RAID 1 with two spinning disks).

Is the fact that Apache is writing its access log to these same disks affecting the speed of reads? Do you typically add a separate spindle for logs? If so, do you add a single spindle or RAID it (you'll obviously lose log data otherwise if the disk fails)?

Or should we be pushing Apache logs over a network interface to a central logging server? I assume probably a separate network interface than the one that is serving all those HTTP requests?

5 Answers 5

4

Having a busy webserver logging to a RAID 1 is problematic. I can't remember at exactly what point we had to change our completely logging/archiving strategy but it was somewhere around a few million hits a day.

I had to migrate the logging to RAID 0 disks and to overcome the possible loss of data, implemented Facebook's newly open sourced Scribe technology to move the logs to a centralized logging server. Now we are now at several hundred million hits per day and we are moving terabytes of logs from our front ends, through scribe, to a central logging server which now makes analyzing those logs, graphing data trends and monitoring much easier. For your purposes a single scribe server would handle that traffic and moving that data easily.

1

Check out: mod_log_spread2 port of mod_log_spread

mod_log_spread is a patch to Apache's mod_log_config, which provides an interface for spread to multicast access logs. It utilizes the group communication toolkit Spread,

And send the logs to a log collector.

0

I would say you have answered your own question. Both of your ideas should improve the performance of the webserver. If you add additional drives for the logs they should be RAID 1.

0

The logging for that many hits will definitely be having an effect, but how much effect depends on the size and amount of the actual requests - if the files being served are large than the logging will be making little difference overall and it is just the act of needing to read the files to serve them that is causing I/O contention. If the content requests tends to follow the usual pattern (many people requesting the latest few files with a relatively small number of requests for older content) then adding more RAM may help by letting the OS hold more of the content in cache. If the requests don't follow a useful pattern like that (i.e. the content requests over any given time period has no chance of fitting in a realistic amount of RAM) then this isn't going to help, of course.

If logging activity is significant in your I/O bottleneck, then moving the logs to another device will certainly help. That device could be another drive/array on the same machine or over the network. Unless you are half way saturating the servers NIC at any point in normal activity (which is unlikely - your external bandwidth will be the bottleneck there) then this will not need an extra NIC unless keeping the logging activity away from the interface the public web service is on makes it easier to secure (or otherwise fits in better with your network arrangement). An unsaturated network link is not going to see latency issues in, or due to, Apache logging activity though if there are times when the link is heavily used for long periods Apache processes may end up blocking for extra time (meaning more RAM being used in busy times) while waiting for log writes to complete.

If the reading of content is as much (or more) of a problem as the writing of logs, then you might consider using RAID0 for the content. You may need to beef up your backup arrangements to copy with the extra risk of a single drive death taking out the entire array. You could mitigate this with RAID10 but that will mean using four drives not 2. RAID5 may be an option too as for reads it will perform similarly to RAID0 and would only need one extra disk, but this is not recommended if the content changes often because RAID5s write performance issues [each block write becomes at least one extra read (the neighbour block(s) and an extra write (for the parity block)] will kick you. Also you are better not having your log files going to RAID5 for the same reason.

In summary: moving the logs to other drives (in the same machine or over the network) may help, as will moving the main content to RAID0, RAID10 or RAID5 (beef up your backup arrangements if using R0, be careful of write performance issues with R5). More RAM may also help by reducing the need for the actual I/O operations if most requests in a given time period are for a subset of your data that is of could-fit-in-RAM-with-a-chunk-to-spare size.

0

Adding separate disks to carry out your logging would a fairly logical step and bound to help to at least some degree with your system, however I would think now is the time to start considering the future.

Look at how your traffic has increased on the site, what sort of pace of growth you've maintained and consider where you're going to be 1 year down the line or even 2. If your website continues to grow in popularity then you're going to need to start considering multiple web servers, redundancy, high availability and so on. For that reason I would suggest you consider using rsyslog (which is in ports as rsyslog3) or similar to handle remote logging of files on a back end server. Then whenever you need to add an additional web server it's a simple task of copying the rsyslog settings over from the main webserver and away you go. It also gives you logs on a hopefully not too contended box that will allow you to carry out much more detailed analysis.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .