3

There's a plethora of benchmarks out there, comparing Apache, nginx, lighttpd, LiteSpeed/ OpenLiteSpeed, Cherokee etc.

Unfortunately, it seems that most of the benchmarks are of the "small static file" type, which is quite useless...who uses a webserver to serve small static files?

How come nobody runs benchmarks for DEFAULT installs of php applications like:

  • WordPress
  • Drupal
  • phpBB
  • Menalto Gallery
  • Magento
  • OpenCart
    etc

This would be a much better indication of what works best under real-usage type conditions, and would give folks a much better sense of the most appropriate server for their needs.

The above popular php applications would probably cover the needs of a huge chunk of webserver users.

Sure, in this kind of benchmark, the performance of MySQL, PHP, etc would come into play and we won't get RAW performance figures for the webserver. But who really cares for RAW numbers anyway? Folks are really only concerned about comparative performance, and that's why they check out benchmarks.

If we start doing this kind of benchmarks, we might finally arrive at the conclusion that server comparisons are moot as there aren't significant performance differences between competing products and that there are other items in the infrastructure/setup that need much more optimization attention.

I am sure that folks who run benchmarks are smart and know all of the above...so how come we still continue getting worthless "small static file" benchmarks?

0

2 Answers 2

6

The most important benchmarks are the ones you take against your application before making some changes, then re-running them, on your own hardware in your own server environment.

This won't help if you're trying to choose whether to use framework X versus framework Y for your application or configuration, unless you are in a position to test your servers with multiple configurations before going into production.

But you're right in that when you look at published benchmarks, there's many factors that can affect the results. Are they worthless? Maybe not completely; if a particular framework or server is performing well in multiple tests, it'll probably perform well for you. If it's slow, it'll probably be relatively slow.

You may still have other factors to consider. What good is a fast framework if you're not experienced with it and you have to roll out a product on a deadline? Will the people maintaining it have experience on that platform, or be able to learn it quickly? Maybe you have in-house experience with a particular language or server that would reduce maintenance overhead or adding modules or features. Maybe you need to have a platform that is actively maintained so you can be confident bugs will be fixed in a timely manner, or when you need help there is an active community to offer advice. The fastest server isn't much help if you can't get it configured and tuned in the first place, after all.

1
  • I agree with the heart of your answer, but I'd say that the most important benchmarks are the performance profile of your app in Production (a continuous and constantly watched set of metrics.)
    – gWaldo
    Sep 13, 2013 at 15:46
4

I think you just answered your own question. Static file benchmarks are easy to set up and easy to measure. Once you add an application stack, you're going from one variable (the web server) to possibly dozens. "What about the PHP configuration? Maybe the MySQL server is starved for RAM? Is the disk configuration correct?" and so forth.

Static benchmarks test one thing: How fast can a given web server service an HTTP request? In general, that's all you want a web server to do. Once you're talking about application servers, the game changes completely.

And to be fair, there are benchmarks for web frameworks and some application servers: http://www.techempower.com/benchmarks/

3
  • Even "static file" benchmarks are measuring more than one factor (the web server, the OS, the filesystem, and the disk subsystem are all involved). The performance of your Linux/EXT3/nginx system with old parallel-ATA disks in a software md RAID will be way worse than AIX/JFS2/nginx with 15K SAS drives in a hardware RAID-1. (Benchmarks are hard, let's go shopping!)
    – voretaq7
    Sep 12, 2013 at 18:29
  • @voretaq7 Understood. There's lots of things involved. So if I have nginx on one set of infrastructure the results may be different compared to nginx on another set of infrastructure. But if the primary purpose of benchmarking is comparison, then this is immaterial, as even though other things matter, it really does amount to an "all else being equal" situation. Or am I missing something major here?
    – OC2PS
    Sep 12, 2013 at 18:57
  • @OC2PS the problem is making "all else" equal -- published comparison benchmarks (like Apache vs nginx) are frequently done on disparate systems, so you have to account for that when comparing them. That's why the most useful benchmarks are the ones you do yourself
    – voretaq7
    Sep 12, 2013 at 19:10

Not the answer you're looking for? Browse other questions tagged .