I am running a memcached service in a linux container pinned to a specific physical core. As expected the CPU increases when the number of requests increase. The requests are sent from a workload generator on another machine. However, for a given number of requests, the amount of CPU used varies in counter intuitive ways under different scenarios. The machine has 4 physical cores(0,1,2,3) and hyperthreading has been disabled.
1) Memcached runs alone on a single core (CPU0)
2) Memcached runs on a single core (CPU0), and a CPU intensive application(does many floating point operations) runs on (CPU3).
As an example, Memcached uses a CPU of 88% for 60000 requests per sec for scenario 1. For the same 60000 requests per sec, Memcached uses a lower amount of CPU of 78% for scenario 2. How is this possible? Why does CPU usage of Memcached drop when a CPU intensive application runs on a different core?
Some interrupt results for each core from mpstat:
scenario1(memcached running alone on CPU0):
CPU usage of Memcached = 88%
Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s BLOCK_IOPOLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
Average: 0 0,00 231,61 0,20 102893,92 0,00 0,00 0,00 4,14 0,18 42,53
Average: 1 1,80 18,33 0,00 0,00 0,00 0,00 0,98 4,25 0,02 15,98
Average: 2 0,00 22,96 0,00 0,00 0,00 0,00 0,00 3,47 0,02 20,49
Average: 3 0,00 20,59 0,00 0,00 0,78 0,00 0,00 3,43 0,02 18,71
Scenario 2: (memcached on CPU0 and CPU intensive application on CPU 3)
CPU usage of Memcached = 78%
Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s BLOCK_IOPOLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
Average: 0 0,00 204,87 0,21 102268,58 0,00 0,00 0,00 2,77 0,40 133,05
Average: 1 0,03 24,94 0,00 0,00 0,00 0,00 0,55 5,56 0,06 23,38
Average: 2 0,00 17,23 0,00 0,00 0,00 0,00 0,00 4,23 0,04 16,12
Average: 3 0,00 250,00 0,00 0,00 0,74 0,00 0,00 2,50 0,00 37,69