Discrepancy in memory usage between `top` and `free`

Question

There is something I don't understand in the way top and free report memory usage. If I run top, and sum the memory usage from the %MEM column, I get a much lower memory usage than what free reports, and I don't understand why!

Below are the outputs of both free and top.

root@zamerli:~# free
             total       used       free     shared    buffers     cached
Mem:        329036     314412      14624          0       7820      97708
-/+ buffers/cache:     208884     120152
Swap:       524280       2660     521620


root@zamerli:~# top -b -n 1 
top - 20:12:45 up  2:19,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 103 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.1%us,  0.4%sy,  0.0%ni, 98.3%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    329036k total,   314636k used,    14400k free,     7820k buffers
Swap:   524280k total,     2660k used,   521620k free,    97708k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                         
    1 root      20   0 19320 1292  976 S    0  0.4   0:00.41 init                                                            
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd                                                        
    3 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/0                                                     
    4 root      15  -5     0    0    0 S    0  0.0   0:00.01 ksoftirqd/0                                                     
    5 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/0                                                      
    6 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/0                                                        
    7 root      15  -5     0    0    0 S    0  0.0   0:00.00 cpuset                                                          
    8 root      15  -5     0    0    0 S    0  0.0   0:00.00 khelper                                                         
    9 root      15  -5     0    0    0 S    0  0.0   0:00.00 async/mgr                                                       
   10 root      15  -5     0    0    0 S    0  0.0   0:00.00 xenwatch                                                        
   11 root      15  -5     0    0    0 S    0  0.0   0:00.00 xenbus                                                          
   13 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/1                                                     
   14 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1                                                     
   15 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/1                                                      
   16 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/1                                                        
   17 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/2                                                     
   18 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/2                                                     
   19 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/2                                                      
   20 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/2                                                        
   21 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/3                                                     
   22 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3                                                     
   23 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/3                                                      
   24 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/3                                                        
   25 root      15  -5     0    0    0 S    0  0.0   0:00.00 kintegrityd/0                                                   
   26 root      15  -5     0    0    0 S    0  0.0   0:00.00 kintegrityd/1                                                   
   27 root      15  -5     0    0    0 S    0  0.0   0:00.00 kintegrityd/2                                                   
   28 root      15  -5     0    0    0 S    0  0.0   0:00.00 kintegrityd/3                                                   
   29 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/0                                                       
   30 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/1                                                       
   31 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/2                                                       
   32 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/3                                                       
   33 root      15  -5     0    0    0 S    0  0.0   0:00.00 kseriod                                                         
   34 root      20   0     0    0    0 S    0  0.0   0:00.00 khungtaskd                                                      
   35 root      20   0     0    0    0 S    0  0.0   0:00.00 pdflush                                                         
   36 root      20   0     0    0    0 S    0  0.0   0:00.15 pdflush                                                         
   37 root      15  -5     0    0    0 S    0  0.0   0:00.25 kswapd0                                                         
   38 root      15  -5     0    0    0 S    0  0.0   0:00.00 aio/0                                                           
   39 root      15  -5     0    0    0 S    0  0.0   0:00.00 aio/1                                                           
   40 root      15  -5     0    0    0 S    0  0.0   0:00.00 aio/2                                                           
   41 root      15  -5     0    0    0 S    0  0.0   0:00.00 aio/3                                                           
   42 root      15  -5     0    0    0 S    0  0.0   0:00.00 jfsIO                                                           
   43 root      15  -5     0    0    0 S    0  0.0   0:00.00 jfsCommit                                                       
   44 root      15  -5     0    0    0 S    0  0.0   0:00.00 jfsCommit                                                       
   45 root      15  -5     0    0    0 S    0  0.0   0:00.00 jfsCommit                                                       
   46 root      15  -5     0    0    0 S    0  0.0   0:00.00 jfsCommit                                                       
   47 root      15  -5     0    0    0 S    0  0.0   0:00.00 jfsSync                                                         
   48 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfs_mru_cache                                                   
   49 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfslogd/0                                                       
   50 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfslogd/1                                                       
   51 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfslogd/2                                                       
   52 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfslogd/3                                                       
   53 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfsdatad/0                                                      
   54 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfsdatad/1                                                      
   55 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfsdatad/2                                                      
   56 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfsdatad/3                                                      
   57 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfsconvertd/0                                                   
   58 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfsconvertd/1                                                   
   59 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfsconvertd/2                                                   
   60 root      15  -5     0    0    0 S    0  0.0   0:00.00 xfsconvertd/3                                                   
   61 root      15  -5     0    0    0 S    0  0.0   0:00.00 glock_workqueue                                                 
   62 root      15  -5     0    0    0 S    0  0.0   0:00.00 glock_workqueue                                                 
   63 root      15  -5     0    0    0 S    0  0.0   0:00.00 glock_workqueue                                                 
   64 root      15  -5     0    0    0 S    0  0.0   0:00.00 glock_workqueue                                                 
   65 root      15  -5     0    0    0 S    0  0.0   0:00.00 delete_workqueu                                                 
   66 root      15  -5     0    0    0 S    0  0.0   0:00.00 delete_workqueu                                                 
   67 root      15  -5     0    0    0 S    0  0.0   0:00.00 delete_workqueu                                                 
   68 root      15  -5     0    0    0 S    0  0.0   0:00.00 delete_workqueu                                                 
   69 root      15  -5     0    0    0 S    0  0.0   0:00.00 kslowd                                                          
   70 root      15  -5     0    0    0 S    0  0.0   0:00.00 kslowd                                                          
   71 root      15  -5     0    0    0 S    0  0.0   0:00.00 crypto/0                                                        
   72 root      15  -5     0    0    0 S    0  0.0   0:00.00 crypto/1                                                        
   73 root      15  -5     0    0    0 S    0  0.0   0:00.00 crypto/2                                                        
   74 root      15  -5     0    0    0 S    0  0.0   0:00.00 crypto/3                                                        
   77 root      15  -5     0    0    0 S    0  0.0   0:00.00 net_accel/0                                                     
   78 root      15  -5     0    0    0 S    0  0.0   0:00.00 net_accel/1                                                     
   79 root      15  -5     0    0    0 S    0  0.0   0:00.00 net_accel/2                                                     
   80 root      15  -5     0    0    0 S    0  0.0   0:00.00 net_accel/3                                                     
   81 root      15  -5     0    0    0 S    0  0.0   0:00.00 sfc_netfront/0                                                  
   82 root      15  -5     0    0    0 S    0  0.0   0:00.00 sfc_netfront/1                                                  
   83 root      15  -5     0    0    0 S    0  0.0   0:00.00 sfc_netfront/2                                                  
   84 root      15  -5     0    0    0 S    0  0.0   0:00.00 sfc_netfront/3                                                  
  310 root      15  -5     0    0    0 S    0  0.0   0:00.00 kstriped                                                        
  315 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksnapd                                                          
 1688 root      15  -5     0    0    0 S    0  0.0   0:00.16 kjournald                                                       
 1781 root      20   0 12900  712  520 S    0  0.2   0:00.07 upstart-udev-br                                                 
 1783 root      16  -4 17196  428  292 S    0  0.1   0:00.06 udevd                                                           
 1954 root      18  -2 17192  432  300 S    0  0.1   0:00.00 udevd                                                           
 1955 root      18  -2 17192  172  168 S    0  0.1   0:00.00 udevd                                                           
 2098 root      20   0  8192  448  444 S    0  0.1   0:00.04 dd                                                              
 2104 syslog    20   0  180m 1268  904 S    0  0.4   0:00.01 rsyslogd                                                        
 2252 root      20   0 18708  964  748 S    0  0.3   0:00.01 cron                                                            
 2277 root      20   0 49072 1100  584 S    0  0.3   0:00.00 sshd                                                            
 2282 root      20   0  5988  548  472 S    0  0.2   0:00.00 getty                                                           
 8523 root      20   0  4004  604  492 S    0  0.2   0:00.00 mysqld_safe                                                     
 8631 mysql     20   0  162m  29m 7092 S    0  9.2   0:00.25 mysqld                                                          
 8632 root      20   0  5176  764  644 S    0  0.2   0:00.00 logger                                                          
10725 root      20   0 28580 1612  868 S    0  0.5   0:00.00 nginx                                                           
10784 root      20   0  107m  33m 3104 S    0 10.6   0:01.14 merb                                                            
10785 root      20   0  135m  38m 3116 S    0 11.9   0:00.88 merb                                                            
10790 www-data  20   0 28580 2208  824 S    0  0.7   0:00.01 nginx                                                           
11022 root      20   0 70440 3232 2488 S    0  1.0   0:00.02 sshd                                                            
11030 root      20   0 19100 2128 1568 S    0  0.6   0:00.02 bash                                                            
11121 root      20   0 19128 1188  876 R    0  0.4   0:00.01 top

Answer 1

Well, the processes don't include memory used by the kernel itself, and its caches etc. %MEM also don't include shared memory, such as shared libraries. So I think that is probably it, nothing to worry about.

As far as what top and free report for totals, they are basically the same, 14,400 free from top, and 14,624 from free. That is probably just the difference from the time difference in the report of the two commands.

Answer 2

In "free"'s output, you should primarily look at the "-/+ buffers/cache" line; the used column on that row will more closely reflect what you'd get by adding up the different "RES" lines from top.

But there's also "Shared" memory. The "SHR" column in top reflects memory that could be shared by that process with another process. In technical terms that will primarily be "mmap"ed files, but in practical terms what that means is mostly libraries. When multiple processes are using the same shared library, the library code will only be loaded in memory once. There's no guarantee that a shared library is actually loaded by another program, however. When a process forks there can be a fair amount of sharing as well. And there's some other ways processes can share memory, but they're likely to be smaller amounts.

Answer 3

Maybe I've misreading the question but

(editted for clarity)

# free
             total       used       free     shared    buffers     cached
Mem:        329036     314412      14624          0       7820      97708

# top -b -n 1 
Mem:    329036k total,   314636k used,    14400k free,     7820k buffers

A tiny difference of less than 200k used could very easily be the difference in how the programs count the memory blocks. It could even be just a difference in processes in the few seconds between the commands.

Edit: sorry I see what you are doing now. Top mem usage does not take into account cached memory, or at least not file system level cached memory in some distros. My guess is top shows somewhere around 60% used right?