Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I understand the various types of cpu usage reported by the top command (6.5%us, 17.2%sy, 0.0%ni, etc...), but why does the total %CPU for each process not add up to any of the Cpu(s) values? For example, below the java process is consuming 77.5% CPU, yet Cpu(s) says 76.0% is still idle. Why is this? This is on a single core system.

top - 05:53:27 up 32 min,  2 users,  load average: 0.16, 0.29, 0.34
Tasks:  71 total,   1 running,  70 sleeping,   0 stopped,   0 zombie
Cpu(s):  6.5%us, 17.2%sy,  0.0%ni, 76.0%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   1758616k total,   643432k used,  1115184k free,    12224k buffers
Swap:   917500k total,        0k used,   917500k free,   304608k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                      
 1307 tomcat    20   0  683m 287m 9528 S 77.5 16.8  10:54.99 java                                                                                                                                                                          
 1571 ec2-user  20   0  2592 1096  872 R  1.0  0.1   0:08.61 top                                                                                                                                                                           
    1 root      20   0  2892 1364 1168 S  0.0  0.1   0:00.28 init                                                                                                                                                                          
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd            

Edit: Turns out this is a dual core system. Here is updated output after pressing 1 while the top command is up:

top - 06:10:21 up 49 min,  2 users,  load average: 0.28, 0.37, 0.34
Tasks:  71 total,   1 running,  70 sleeping,   0 stopped,   0 zombie
Cpu0  :  9.9%us, 19.7%sy,  0.0%ni, 69.0%id,  0.0%wa,  0.0%hi,  1.4%si,  0.0%st
Cpu1  :  5.0%us, 10.0%sy,  0.0%ni, 85.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1758616k total,   677548k used,  1081068k free,    13296k buffers
Swap:   917500k total,        0k used,   917500k free,   305732k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                      
 1307 tomcat    20   0  683m 318m 9528 S 68.0 18.6  17:23.53 java                                                                                                                                                                          
    1 root      20   0  2892 1364 1168 S  0.0  0.1   0:00.28 init                                                                                                                                                                          
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                                                                                                      
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.91 ksoftirqd/0                                                                                                                                                                   
    4 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0   
share|improve this question
    
Press the 1 (number one) key while running top and paste the result again. –  Michael Hampton Dec 27 '13 at 6:09
    
I updated the post after pressing 1 while running top. –  OpenUserX03 Dec 27 '13 at 6:12
    
    
Hmm, still seems pretty far off to me if Cpu0 is reporting 69% idle and Cpu1 is reporting 85% idle. That should mean total CPU time from all processes shouldn't be more than 46%, yet java is consuming 68%. That's 22% not accounted for. –  OpenUserX03 Dec 27 '13 at 7:34
    
I hope this isn't a virtual machine; if it is, all bets are off since the stats may not be reported correctly in any case. –  Michael Hampton Dec 27 '13 at 7:37

2 Answers 2

up vote 0 down vote accepted

Those two information that you are comparing won't match, simply, because they are collected from different files. That is although top shows the information in the same terminal, they aren't collected from the same source.

I simply ran an strace on top (running in batch mode). This is where it shows the system wide CPU information.

16:04:04.081092 open("/proc/stat", O_RDONLY) = 6 <0.000022>
16:04:04.081154 lstat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0 <0.000015>
16:04:04.081211 lstat("/proc/stat", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0     <0.000013>
16:04:04.081267 fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000013>
16:04:04.081334 fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000013>
16:04:04.081385 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f699ace2000 <0.000016>
16:04:04.081440 lseek(6, 0, SEEK_SET)   = 0 <0.000013>
16:04:04.081494 read(6, "cpu  302573 6910 83103 10092403 "..., 1024) = 1024 <0.000070>
16:04:04.081656 write(1, "%Cpu(s):  2.9 us,  0.8 sy,  0.1 "..., 80) = 80 <0.000034>
16:04:04.081763 write(1, "KiB Mem:   8048484 total,  41402"..., 73) = 73 <0.000035>
16:04:04.081858 write(1, "KiB Swap:  8060924 total,       "..., 72) = 72 <0.000034>
16:04:04.081940 write(1, "\n", 1)       = 1 <0.000026>

Now, if you see /proc/stat, it shows all the CPUs of the system. top knows that too, because before opening /proc/stat, it opens sys filesystem.

16:04:03.367339 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3 <0.000027>
16:04:03.367408 read(3, "0-7\n", 8192)  = 4 <0.000019>
16:04:03.367464 close(3)                = 0 <0.000015>

Now, when it comes to collecting individual process information, it gets that from /proc/pid/statm and /proc/pid/stat file. (replace pid with well, actual pid).

As you can see, /proc/stat is system wide information for ALL the CPUs and individual proc files for the pids are their specific pid-only information.

So, they won't match.

share|improve this answer

Sampling, which is how top measures CPU use, is subject to error.

The best way to explain it is like this: Imagine a factory that produces exactly one car per hour, on the hour. Say you decide to sample the rate at which the factory produces cars. You start sampling at 5:59 and stop sampling at 7:01. You see two cars produced, one at 6:00 and one at 7:00. You sampled for 62 minutes and 2 cars were produced. Thus you calculate that the factory was producing cars at about 200% of its rated capacity.

In addition, you cannot compare top values against each other because top doesn't provide you a set of measurements of a single system state but a set of independent measurements each subject to their own set of conditions.

For example, the per-CPU values can be computed using a completely different mechanism from the per-process values. The per-CPU values can be exponentially decayed while the per-process values can be the difference between two totals. So they can reflect measurements of the same type of thing, but using completely different methodologies.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.