Best Practices for Webserver Benchmarking

Question

I have a webserver that I wish to benchmark before I make some optimizations to it to see if they have any effect.

However, I want to know what are the best practices for benchmarking?

For example, a co-worker told me to benchmark the machine with another machine on a local network to eliminate network traffic problems.

However, I was thinking of using an off-site machine to benchmark because I wanted to see if the optimizations would make any difference in a real-world case.

I was arguing that many of the speed tweaks deal with optimizing network connection. For example, in Apache, the KeepAlive value allows browsers to use a single TCP connection to request multiple objects, instead of opening and closing a connection for each resource.

If the test was done on a local network connection, then that tweak wouldn't really make that much of a difference, right? The same with minimizing js/css and removing whitespace/comments from HTML.

On the other hand, I do see the problem with internet traffic making each time the benchmark run from being consistent. I wouldn't really know if the changes in the numbers are from the tweaks or if the servers in between had increased or decreased loads.

Who's right? What is the best practice for benchmarking? Should we do both?

tl;dr - Should I benchmark using a machine on the local network/off-site/or both?

Answer 1

This all depends very much about your site. Doing both off-line and on-site benchmarks won't hurt. But, what to tweak and how to test?

If you are mostly serving out static content or some very simple dynamic content and doing so at an insane rate, then tweaking things in Apache timeout / kernel-level makes sense.

If you have a dynamic site, then tweaking very soon becomes a different beast. Sure, making all those Apache and kernel level optimizations are still a sensible thing to do, but don't get fooled and smoke-mirrored by those.

When it comes to performance tweaking / testing, always first take care of the slowest parts. With a dynamic site the real bottleneck is most likely 1) the database or 2) the scripts your site is running.

If the slowest part is the database, it doesn't make sense to performance-tune your Apache before you can make your database fast. In addition to that you'll need to be sure you have proper caching techniques in place and you run memcached or similar if possible.

So, that was about what to benchmark and tweak. Now, how to benchmark?

I like to run two kinds of benchmarks. If the numbers are already known -- say, a site is getting a revamp, so you know unique visitors, page loads and similar statistics from the old site --, I run a realistic benchmark with rate similar to current web site. I also benchmark it with a similar rate * 2 to see if it can stand a little bit of future growth.

The other kind of benchmark I like to run is to smoke and burn the site. I'll just torture the hell out of it by running benchmark as fast as possible with insane amounts of simultaneous requests. If it can stand it, great, if it cannot, I find what's the breaking point.

Tools I like: ab, siege, JMeter.

Answer 2

I have used a custom log format with the %T (service time) specified on an ongoing basis. se I replaced %l (identd name) in the format. This can be used to identify slow pages for tuning. Large responses on slow lines can cause false positives, as can network retries.

I used a custom log summary script to identify pages with slowest responses and average response time. These may be good targets for optimization. Comparing reports over time helps identify upcoming problems.

Client side tools can provide good stress benchmarks and verify fixes haven't broken the site. There are factors which can provide benchmarks which do not match what you get in production.