I've been rethinking the compiler shootout and I began to wonder: is Amazon EC suitable for benchmarking? The environment is totally virtual, so are there any guarantees that benchmarking results will be repeatable? Or should I assume that the virtual hosting environment time slicing interferes with timing approaches?



Not that this is EC2's fault, but when benchmarking, when comparing A and B, one needs to keep in mind the phrase "all things being equal". (There is a formal name for this, but I forget.) With the EC2 virtualized environment, where you are not in control of anything but your instance, this is just not the case.

Unless you are benchmarking cloud environments ;)

(Edited to be specific to EC2, thanks to knowledgeable comments)

  • s/visualized/virtualized – jldugger May 7 '09 at 18:50
  • I don't think that cloud computing necessarily means its impossible to offer strict guarentees about availability. Probably the main challenge comes from NOT giving a CPU intensive VM the slack time. – jldugger May 7 '09 at 18:53
  • 1
    Fair enough, folks. I will qualify my answer as EC2 specific. I've done some benchmarking on EC2, mostly bandwidth and response time oriented, and was surprised at how variable things are. All things are not equal. – Stu Thompson May 7 '09 at 19:01
  • 1
    @jldugger: exit 0 – Stu Thompson May 7 '09 at 19:07

In benchmarking, you want to control all of the variables you can to ensure proper testing. In a cloud computing environment, I would not trust that a given instance will be exactly the same from one moment to the next enough to perform proper benchmarking reliably.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.