I like to fully-load our compute hardware to reduce wasted CPU time, and on typical in-house hardware this is fairly easy: load the machine with as many runnable threads as there are cores, and idle time will go to zero.
Here is an example app:
public class Looper
{
public static void main(String[] args)
{
while (true) { new java.util.Random().nextBytes(new byte[4096]); }
}
}
On our in-house, 8-core hardware, I can run 8 of these and idle time (as reported by mpstat and top) goes to zero. I can even add a 9th, 10th, etc process, and idle time stays very close to zero.
On EC2 (c1.xlarge instances), however, idle time is much higher than I'd expect. At 8 processes, idle time hovers around %1, and with 9, 10, etc processes, it can increase to 2%-3% or higher. With more complicated programs (not the example above), idle time can be even higher than that.
Can anyone explain this? This is with very recent Amazon kernels, and does not include stolen CPU time, which I would expect to see on EC2. Is this a problem with EC2 in particular, or is it general to Xen? Are there known workarounds?
