Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've been rethinking the compiler shootout and I began to wonder: is Amazon EC suitable for benchmarking? The environment is totally virtual, so are there any guarantees that benchmarking results will be repeatable? Or should I assume that the virtual hosting environment time slicing interferes with timing approaches?

share|improve this question

2 Answers 2

up vote 1 down vote accepted

No.

Not that this is EC2's fault, but when benchmarking, when comparing A and B, one needs to keep in mind the phrase "all things being equal". (There is a formal name for this, but I forget.) With the EC2 virtualized environment, where you are not in control of anything but your instance, this is just not the case.

Unless you are benchmarking cloud environments ;)

(Edited to be specific to EC2, thanks to knowledgeable comments)

share|improve this answer
    
s/visualized/virtualized –  jldugger May 7 '09 at 18:50
    
I don't think that cloud computing necessarily means its impossible to offer strict guarentees about availability. Probably the main challenge comes from NOT giving a CPU intensive VM the slack time. –  jldugger May 7 '09 at 18:53
    
1  
Fair enough, folks. I will qualify my answer as EC2 specific. I've done some benchmarking on EC2, mostly bandwidth and response time oriented, and was surprised at how variable things are. All things are not equal. –  Stu Thompson May 7 '09 at 19:01
1  
@jldugger: exit 0 –  Stu Thompson May 7 '09 at 19:07

In benchmarking, you want to control all of the variables you can to ensure proper testing. In a cloud computing environment, I would not trust that a given instance will be exactly the same from one moment to the next enough to perform proper benchmarking reliably.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.