I couldn't find this documented elsewhere. Looking into the code:
There are two modes for CPU metrics reporting: the default one, and a "detailed CPU time" which can be enabled from the Setup screen (Display Options / Detailed CPU time). All of them show the percentage of time spent in different processes:
Blue: low priority processes (nice > ...
Your server is within some kind of virtuozzo/openvz/virtualization-du-jour container. Trying to make sense of memory usage is tilting at windmills.
Linux ate your RAM! But that's okay, it does it to everyone.
The Long Story
Let's break it down!
In the Mem: section we have:
$n total: the amount of physical RAM in your machine
$n used: ...
The data exposed by top is often insufficient or misleading in virtualized environments like Amazon EC2 and the reported percentage depends on your instance type and the underlying processor core utilization (which usually doesn't match the virtualized hardware you are presented with from the hypervisor), amongst other things - what you are seeing is most ...
Running top in batch mode to report memory sizes periodically can be used to see who is using the memory when things go south. Runing sar in batch mode should give some good diagnostics on memory use, and related I/O. Running munin to monitor the system should give you a graph with good detail on what memory is being used for. This may help a lot.
I found this explanation from Mugurel Sumanariu very clear:
VIRT stands for the virtual size of a process, which is the sum of
memory it is actually using, memory it has mapped into itself (for
instance the video card’s RAM for the X server), files on disk that
have been mapped into it (most notably shared libraries), and memory
shared with other ...
A very good observation and we have run into this as well. Here's what I found:
Be careful measuring CPU usage from within an EC2 instance. It’s possible to see CPU usage well below 100%—and yet be completely maxed out. Trust me: been there, done that. (CloudWatch CPUUtilization, by the way, is measured from outside the instance and is always correct.)
The correct answer is: -U '!root' (or -u '!root' on some Ubuntus). This was introduced in top v3.2.9:
man top for -U option:
Prepending an exclamation point ('!') to the user id or name instructs top to display only processes with users not matching the one provided.
Remember to put the exclamation mark and username in single quotes.
You can use this command to see the top 10 applications regarding RAM usage:
ps -A --sort -rss -o comm,pmem | head -n 11
Sometimes this command helps you if many sub processes have been generated:
This way you can see which processes belong together.
It's normal if you have a multi-threaded process on a machine with more than one core/thread/processor.
Abstract of man top:
In a true SMP environment, if a process is multi-threaded and top is not operating in Threads mode, amounts greater than 100% may be reported.
Your smem -tw output shows that your kernel is consuming over 9 GB of dynamic memory:
danslimmon@bad-server:~$ smem -tw
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 12857576 2887440 9970136
The column displays CPU time spent per process rather than real time. I can't see if this is specified in the man page but here is some copy-pastage from about.com:
Total CPU time the task has used since it started. If cumulative mode is on, this also includes the CPU time used by the process's children which have died. You can set cumulative mode ...
MongoDB uses memory mapped files - which can be much larger then the actual physical (or swap) available on your machine. The total virtual memory used will usually be the size of all the data on disk at the time as a result.
The resident memory will be the stat that represents the actual working set used, though depending on your resources on the box ...
You have about 1692m in free memory according to Linux.
In the top "Mem" line it would seem like your memory is almost 99% (7793m / 7840m) but in reality you're only using about 78% of available memory. See also http://www.linuxatemyram.com/ for a more in-depth explanation.
Free vs Top vs /proc/meminfo
Adding up the memory used by ...
About 30% of CPU time seems to be spent in WAIT, and your (1 minute, at least) load is very high.
So I would start by checking your storage, and storage usage patterns.
A good starting point might be taking a look at iostat and/or monitoring for any slow ops (writes and reads).
You might also check the time spent on each process in top, to see if anything ...
Nothing is really using that memory in terms of applications.
You need to deduct the 'cached' value which represents the page cache to get a better idea as to what your actual memory usage is in terms of program usage.
Basically this is good memory management and this is ideally what you want.
See the link here for more infomation: http://www....
You can try something like:
TERM=vt100 top -b -n 1
This will set the TERM variable to execute with the "top" command.
But your underlying problem is that your termcap database is incomplete or broken. I don't have a Debian machine to look at, but on an Ubuntu box, the "dumb" termcap info is in the file /usr/share/terminfo/d/dumb. This file is part of the ...
Well, in case you're feeling brave:
gdb -p 20788
then issue bt to see the stack-frame, for e. g.
And BTW, there's also ltrace to mention — try it as well.
UPD.: well, ok, since now we have an idea that Apache is really running something, why wouldn't ya look at mod_status output — Extended one?
There is top's secure mode, when it is invoked as
In that mode the user can't change the refresh delay of top, kill or renice processes.
If you want to make it system wide instead of on a per-invocation basis, you may use the /etc/toprc file with the following contents:
Only two lines: first one to set secure-mode, and the second to set the ...
One possibility is /proc mounted with either hidepid=1 or hidepid=2. This mount option was added in latter Linux kernels and back ported sometime around CentOS 5.9 and 6.3.
The proc filesystem supports the following mount options:
hidepid=n (since Linux 3.3)
This option controls who can access the information in
Load is a measure of the workload a system has had on a 1, 5 and 15 minute basis.
The most common misconception is that Load Average is purely connected to the CPU usage of a system.
Load does however incorporate additional measurements such as CPU waiting for I/O which I think is your issue.
Based on the image I'm guessing you ran out of memory and ...