Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I had several times received notifications from my monitoring system about 98% memory. I ran top and counted only around 60% memory used if I sum memory column. After several hours memory usage was back to normal (~70%). I suspected fs cache, but free -m does not prove this.

Any ideas? Server: x86-64, ubuntu 12.04, 8Gb RAM

# free -m

             total       used       free     shared    buffers     cached
Mem:          7958       7835        123          0          8         39
-/+ buffers/cache:       7787        171
Swap:          975        975          0

#top

top - 10:11:48 up 179 days, 14:25,  1 user,  load average: 1.99, 1.81, 1.59
Tasks: 143 total,   1 running, 142 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.8%us,  0.6%sy,  0.0%ni, 82.8%id, 13.6%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   8149424k total,  8017556k used,   131868k free,     6708k buffers
Swap:   999420k total,   999372k used,       48k free,    37952k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND        
30562 mysql     20   0 6033m 2.6g 3160 S   14 32.9  74:37.56 mysqld         
10898 www-data  20   0 25.3g 2.0g 1560 S   16 25.6  19:44.10 java           
 9570 rabbitmq  20   0 2611m 166m 1540 S    6  2.1 628:08.96 beam.smp       
13166 redis     20   0 94276  72m  584 S    0  0.9  23:24.47 redis-server   
 3798 root      20   0  348m  29m  988 S    0  0.4 167:36.37 cimprovagt     
 7614 root      20   0 51468  28m 1256 S    0  0.4   0:02.59 bash           
 3677 root      20   0 4706m  26m    0 S    0  0.3 110:47.59 java           
 3458 bind      20   0  668m 5328 1488 S    0  0.1   0:23.38 named          
 7595 root      20   0 85380 2956  880 S    0  0.0   0:00.25 sshd           
 2061 root      10 -10  4996 2948 2104 S    0  0.0  13:39.72 iscsid         
  632 syslog    20   0  244m 2404  360 S    0  0.0   7:12.39 rsyslogd       
 5855 newrelic  20   0  107m 1852 1248 S    0  0.0 128:23.28 nrsysmond      
    1 root      20   0 24340 1516  736 S    0  0.0   0:29.96 init           
 2741 root      20   0 17340 1308  944 R    0  0.0   0:00.04 top            
 4214 root      20   0 45336 1264 1020 S    0  0.0   6:12.72 cdm            
 4235 root      20   0 12428 1108  908 S    0  0.0  10:55.38 processes      
 4178 root      20   0 19124 1064  856 S    0  0.0   7:20.07 controller     
 5853 ntp       20   0 37772 1036  872 S    0  0.0   7:40.35 ntpd           
 4179 root      20   0 86044 1024  848 S    0  0.0   9:55.54 spooler        
 4008 root      20   0 4090m  980  756 S    0  0.0   0:08.70 console-kit-dae
 3790 root      20   0  422m  952  668 S    0  0.0  89:21.80 cimprovagt     
 3629 root      20   0 50032  868  752 S    0  0.0   1:17.44 sshd           
 4075 root      20   0  182m  844  408 S    0  0.0   0:02.90 polkitd        
27661 root      20   0  101m  776  772 S    0  0.0   0:00.02 mysql          
18465 root      20   0  101m  732  732 S    0  0.0   0:00.04 mysql          
 3436 root      20   0 19112  716  636 S    0  0.0   0:11.25 cron           
 3783 root      20   0 1009m  716  432 S    0  0.0   0:04.83 cimserver      
26239 root      20   0 22092  700  696 S    0  0.0   0:00.03 bash           
 4181 root      20   0 13196  696  596 S    0  0.0   4:18.57 hdb            
 3406 root      20   0 14744  664  660 S    0  0.0   0:00.00 getty          
 3410 root      20   0 14744  664  660 S    0  0.0   0:00.00 getty          
 3428 root      20   0 14744  664  660 S    0  0.0   0:00.00 getty          
 3429 root      20   0 14744  664  660 S    0  0.0   0:00.00 getty          
 3432 root      20   0 14744  664  660 S    0  0.0   0:00.00 getty          
 3972 root      20   0 14744  664  660 S    0  0.0   0:00.00 getty          
18452 root      20   0 22076  620  620 S    0  0.0   0:00.00 bash           
  361 root      20   0 22104  568  568 S    0  0.0   0:00.04 udevd          
 3441 root      20   0 15980  560  464 S    0  0.0  26:28.16 irqbalance     
  654 messageb  20   0 23948  516  324 S    0  0.0   0:01.98 dbus-daemon    
  676 root      20   0 21188  504  504 S    0  0.0   0:00.00 bluetoothd     
 3665 root      20   0  4400  492  488 S    0  0.0   0:00.00 sh             
 5546 rabbitmq  20   0  7672  488  308 S    0  0.0   1:04.49 epmd           
26238 root      20   0 26092  476  476 S    0  0.0   0:00.24 screen         
 2060 root      20   0  4504  440  400 S    0  0.0   3:18.60 iscsid         
share|improve this question
1  
Your data show that you have truly used up all of your memory and swap. It's time for an upgrade. –  Michael Hampton Jan 13 at 16:32
    
But it does not most of the time. It just sometimes goes crazy and I try to understand who is responsible for that. –  Daniil Jan 13 at 16:35
1  
@MichaelHampton, so you're telling everything else except the first two mysqld and java are taking up to ~ 3.5 GiB or RSS, aren't you? ) The output is sorted by RSS, BTW. –  poige Jan 13 at 16:36
1  
@poige No...you need to look at other stuff, like slabtop. –  Michael Hampton Jan 13 at 16:37
4  
@poige My money is on the Java web app misbehaving. Who writes anything important in Java anyway? –  Michael Hampton Jan 13 at 16:39

2 Answers 2

buffers is the FS cache.

Buffers are associated with a specific block device, and cover caching of filesystem metadata as well as tracking in-flight pages. The cache only contains parked file data. That is, the buffers remember what's in directories, what file permissions are, and keep track of what memory is being written from or read to for a particular block device. The cache only contains the contents of the files themselves.

http://stackoverflow.com/a/12547130/636573

Top also shows 13% of IOWAIT on the snapshot you gave us. To me this looks like a OLTP database not getting nearly enough of the disk IOPS it needs. Upgrade your storage subsystem.

share|improve this answer
    
Yup, but I still can't understand who ate rest of my 30% of memory: buffers are just 8MB, -/+ buffers/cache: 7787 which is not 8GB * 60% = 4.8GB –  Daniil Jan 13 at 17:05
    
@Daniil Sorry, you're right, I was looking at the wrong column. Still, 100% of your swap space is used and nearly 100% of your RAM is used. It's clear that you're using more than you have. You either have to tune the MySQL server/Java applet to not use more than X amount of memory or upgrade. The reason you're seeing a high IOWAIT is because of the swapping. –  pauska Jan 13 at 17:15
    
@pauska, there's nothing clear except you're mistaken, actually. ) Dude needs slabtop and I'd like to look at it as well. –  poige Jan 13 at 17:27
    
@pauska upgrade costs money and I first would like to know if it is really necessary or I can do something without it –  Daniil Jan 13 at 17:33

OK, first go back and make sure your top output is sorted by by memory use (RESident size), and if it's not go ahead and update your question with that output. Then you'll actually be able to see what's chewing up your RAM.

Second, forget the %MEM column. That particular cake is a lie (due to rounding).
Concentrate on the RESident size of programs (and if you're concerned about swap to, the VIRTual size) instead - sum those columns and the numbers will match what you're seeing in free & top output.


From looking at your top output you have a giant (2.6G) MySQL process, and a giant (2.0G) Java process -- I suspect these two are your co-conspirators in chewing up all your RAM (whatever that particular Java process is asking MySQL to do is generating huge result sets or intermediate data).
The Java process also has a 25G virtual size (!!) - clearly it's leaking memory internally that the garbage collector isn't freeing (or perhaps it's dealing with a huge result set, inefficiently).

I would wager that your system probably cruises along using 4-5G of RAM on a "normal day", and when these two processes get together they chew up the rest (and then some) and you wind up in a Bad Place.
Find out what they're doing, fix it, and your problem will go away.

share|improve this answer
    
As you can see my top result is already sorted by RES and summarizing it up gives me less than a 5GB instead of 7.8GB eaten. I know that MySQL+Java are heavy, but that does not answer who ate rest of my memory and why this problem occurs eventually while memory usage by processes had been nearly the same. –  Daniil Jan 13 at 17:10
    
By the way, 25GB virt size is really strange since java is limited with -Xmx=2500M –  Daniil Jan 13 at 17:14
1  
@Daniil ignore the VIRT column, it doesn't relate to used physical memory. –  mtm Jan 13 at 17:20
1  
"have a giant (2.6G) MySQL process, and a giant (2.0G) " — 4.6 G "giant" in 2014? ORLY :) –  poige Jan 13 at 17:29
    
even RES can't be entirely trusted... but in your case where you have single processes taking huge amounts of memory the % error is small. It fails if you are totalling the RES of an apache httpd processes with 100 children in which you get a huge error due to shared memory. –  mtm Jan 13 at 17:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.