3

how can we determine %sy process that is Consuming CPU.

In the following case there is no netfilter and traffic is under 1 mbps.but still system process are taking too much cpu and cpu usage for niced process are also high.How can we determine the process consuming CPU from system level. 

top - 01:22:18 up  10:09,  3 users,  load average: 14.36, 13.68, 11.68
Tasks: 200 total,   3 running, 197 sleeping,   0 stopped,   0 zombie
Cpu0  :  3.1%us, 63.5%sy, 33.3%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  1.8%us, 34.2%sy, 64.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16436984k total,  8449956k used,  7987028k free,    73420k buffers
Swap:  8385920k total,        0k used,  8385920k free,  5566404k cached
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
5706 tlmsys    27   4 4490m 1.6g  13m S 51.6 10.2  12:37.67 java
4233 oracle    25   0 3448m  47m  33m R  9.9  0.3   0:03.62 oracle
3166 root      15   0 62616 1216  656 S  0.7  0.0   0:01.47 sshd
5512 tlmsys    15   0 96992  12m 9424 S  0.7  0.1   0:04.03 stlfetch
5520 oracle    15   0 3424m  72m  69m S  0.7  0.5   0:08.42 oracle
6 root      10  -5     0    0    0 S  0.3  0.0   0:01.59 events/0
4476 monitor   15   0 90116 1764 1008 S  0.3  0.0   0:00.61 sshd
5872 tlmsys    25   4 4479m 135m  11m S  0.3  0.8   0:25.72 java
7139 oracle    16   0 12740 1180  824 S  0.3  0.0   0:06.76 top
9268 root      16   0 12740 1180  816 S  0.3  0.0   0:02.80 top
9978 root      15   0 12740 1176  820 R  0.3  0.0   0:00.07 top
1 root      15   0 10348  696  584 S  0.0  0.0   0:00.79 init
2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.02 migration/0
3 root      34  19     0    0    0 S  0.0  0.0   0:00.03 ksoftirqd/0
4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.01 migration/1
5 root      34  19     0    0    0 R  0.0  0.0   0:00.00 ksoftirqd/1
7 root      10  -5     0    0    0 S  0.0  0.0   0:00.16 events/1
8 root      10  -5     0    0    0 S  0.0  0.0   0:00.09 khelper
145 root      12  -5     0    0    0 S  0.0  0.0   0:00.03 kthread
150 root      10  -5     0    0    0 S  0.0  0.0   0:00.15 kblockd/0
151 root      10  -5     0    0    0 S  0.0  0.0   0:00.03 kblockd/1
152 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid
311 root      13  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/0
312 root      13  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/1
315 root      13  -5     0    0    0 S  0.0  0.0   0:00.00 khubd
317 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod
389 root      18   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
390 root      15   0     0    0    0 S  0.0  0.0   0:00.78 pdflush
391 root      13  -5     0    0    0 S  0.0  0.0   0:00.00 kswapd0
392 root      13  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
393 root      13  -5     0    0    0 S  0.0  0.0   0:00.00 aio/1
599 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kpsmoused
645 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 mpt_poll_0

its clear that java is taking 51.6 %cpu and oracle 9.9 so even if all the process cpus are combined its under 100% and load average should be under 2,

but why load average is 14.

How can we see kernel process (%sy) which is using high cpu.

uname -a

Linux 2.6.18-155.el5 (todbase1)

if it is kernel bug.Can't we still see the process from system side.

Improve this question
3
  • 1
    Load average and CPU utilization aren't always linked. I don't feel we have enough information here.
    – ewwhite
    Jun 17, 2013 at 7:28
  • You have a pretty old kernel there. Current EL5 should be 2.6.18-348.6.1 (RHEL 5.9). Is there anything special about RAM or network in your environment?
    – Nils
    Jun 17, 2013 at 13:30
  • Could you please specify what you mean by "special about RAM or network".Network load is normal and RAM is 16 Gb
    – Kevin Parker
    Jun 17, 2013 at 13:45

3 Answers 3

Reset to default
5
+50

%sy indicates both the time spent in kernel processes (really kernel threads), and those parts of the kernel which are not processes. Kernel processes show up normally in top (ksoftirqd/0 is one). If you are sorting by instantaneous CPU usage, and it is a kernel process contributing to %sy, that process would show up. However, if it is not a kernel process but the kernel that is executing, top will not provide any information.

Specifically, the kernel effectively has an interrupt context (really several of them, for hardware and software interrupts), and a process context (for running system calls). If memory serves, the latter type will show up as CPU time by the process which caused them. However, the scheduling aspects of the kernel, and the activity of kernel components not related to system call or other interrupt processing, will not have detailed accounting information.

Improve this answer
2
  • What do you make of these high nice-values?
    – Nils
    Jun 17, 2013 at 13:21
  • 1
    They look normal to me.
    – Falcon Momot
    Jun 25, 2013 at 7:24
1

For system-time there are two major IO-candidates:

  • Network - but your buffers are low
  • Disk-IO - normally you should see wait, too - not the case here
  • minor candidate might be RAM throughput (java garbage collection?)

What puzzles me a bit is your high nice value.

Perhaps a cat /proc/interrupts will give you more information here.

I tend to say that you have network related problems.

So add the output of ethtool -S IFACE |grep -vw 0, too. Replace IFACE with the name of your main NIC.

Improve this answer
2
  • I dont have those data any more,a restart brought things under control. over all traffic rx/tx were 8mb and 60 mb.
    – Kevin Parker
    Jun 17, 2013 at 13:51
  • Both java and oracle are running with lowered priorities, hence the nice.
    – Chris S
    Jul 29, 2013 at 13:38
1

With newer kernel you can use perf top and on redhat 5.X if you would like to look in the deep you can use systemtap, be careful with systemtap on live production system, because it work in kernel space

Improve this answer

Not the answer you're looking for? Browse other questions tagged .