I've been rethinking the compiler shootout and I began to wonder: is Amazon EC suitable for benchmarking? The environment is totally virtual, so are there any guarantees that benchmarking results will be repeatable? Or should I assume that the virtual hosting environment time slicing interferes with timing approaches?
|
feedback
|
|
No. Not that this is EC2's fault, but when benchmarking, when comparing A and B, one needs to keep in mind the phrase "all things being equal". (There is a formal name for this, but I forget.) With the EC2 virtualized environment, where you are not in control of anything but your instance, this is just not the case. Unless you are benchmarking cloud environments ;) (Edited to be specific to EC2, thanks to knowledgeable comments) | |||||||||||||||
feedback
|
|
In benchmarking, you want to control all of the variables you can to ensure proper testing. In a cloud computing environment, I would not trust that a given instance will be exactly the same from one moment to the next enough to perform proper benchmarking reliably. | |||
|
feedback
|
