I am running some experiments on a research database on EC2 using a c1.xlarge instance. As far as I can tell, the c1.xlarge uses 8 hyper-threaded virtual CPUs. Amazon also says that this instance uses a physical processor from the "Intel Xeon Family".

Again, the system has 8 cpus, 4 workers and 4 others (lock manager, communicator, 2 sequencers). Therefore we have a 1 to 1 thread to CPU correspondence. The experiment being run is on checkpointing. We are creating a 9th thread to take a checkpoint and evaluating its effect on throughput. The results are predictable when this 9th thread is assigned to one of the CPUs containing one of the 4 workers or lock manager - the throughput drops. However, when we place this 9th thread on one of the remaining cpus, we expect not to see any effect since these threads are not bottlenecks of the system and should not effect the worker threads. However, I am seeing throughput drop and I am searching for an explanation.

I have begun to suspect that there may be a problem if the checkpointing thread spawned on a non-worker thread is somehow invalidating the cache's of the worker threads. I dont have a good understanding of how the caching works on this particular EC2 instance or on Intel Xeon Processor and so I am looking for an explanation as to how this caching works, in particular how it works in a hyper-threaded system. Do both cpus on a hyperthreaded processor share cache? Do threads share cache across processors?

I was able to find an Intel manual which says that the intel Xeon processor 3000 and 5000 series use a "smart second level cache that enables data sharing between two cores to reduce memory traffic". Is this what the instance might be using, and if so does that mean all 8 vCPU's share cache?

  • It is the nature of hyperthreading that increasing the load on one virtual core decreases the resources available to the other virtual core. Performance of a thread, even with a dedicated virtual core, can be impacted by increase utilization of the phsyical core by the other virtual code that shares it. And with virtualization, that other virtual core may even be running some other VM's thread. (Think about it -- hyper-threading gives you twice as many threads but only 15% more performance. Therefore the performance per thread must be much less.) – David Schwartz Dec 6 '13 at 23:25

Since there is a virtualization layer between your server OS and the hardware with EC2, there's not even a guarantee that all 8 threads from your vCPUs are executing on the same physical CPU. Trying to measure things like cache hits or patters from inside the guest are an exercise in futility. You don't have visibility into the actual hardware.

A vCPU doesn't represent a physical CPU core, and 8 vCPUs doesn't represent a single physical CPU with 8 logical cores. Certainly, the CPU scheduler for most hypervisors will try to schedule threads from the same guest to execute on the same core in a multi-processes system, but there is no guarantee.

| improve this answer | |
  • Would you mind providing a little more explanation? I am still confused since Amazon states that the c1.xlarge instance has a "Instance Storage (GB)" of 4x440. I thought this meant that the instance had 4 cores (physical CPUs, correct?), each with 2 virtual CPUs. Does your response suggest that there is no guarantee that any core has 2 vCPUs? To clarify, your response also suggests that it is impossible to say if any of the virtual CPUs are sharing caches? Thanks. – user2713719 Dec 6 '13 at 22:53
  • 2
    You seem a bit confused. The current Amazon instance description for a c1.xlarge is c1.xlarge: 7 GiB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform so you have 8 virtual cores. These don't map to a physical or logical core, they simply enable your VM to schedule that many concurrent threads. Which physical or logical cores those threads are executed on at the physical CPU is up to the hypervisor's CPU scheduler. – MDMarra Dec 6 '13 at 22:59
  • 2
    In short, you cannot be sure that all 8 threads are executing on the same physical CPU and you have no visibility into the physical host's CPU cache from the guest. – MDMarra Dec 6 '13 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.