I am using Ubuntu 10.04 where a number of background jobs run every day. free and top both report that 3.9GB of the 4GB of memory is in use (of which ~90MB is cache/buffers).
When I do a top of ps aux and count the memory usage of my apps, I get to about 50%.
I am only running mysql, apache+passenger, and redis on the machine. It also hosts an NFS share.
Is there another way to check the remaining 49% of in use memory? And to free it without rebooting the server?
This is the output of free:
total used free shared buffers cached
Mem: 4010060 3820592 189468 0 84208 194168
-/+ buffers/cache: 3542216 467844
Swap: 741372 741292 80
ps aux:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 23864 472 ? Ss Feb02 0:13 /sbin/init
root 2 0.0 0.0 0 0 ? S Feb02 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Feb02 2:32 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S Feb02 0:14 [migration/0]
root 5 0.0 0.0 0 0 ? S Feb02 0:00 [watchdog/0]
root 6 0.0 0.0 0 0 ? S Feb02 0:25 [migration/1]
root 7 0.0 0.0 0 0 ? S Feb02 1:30 [ksoftirqd/1]
root 8 0.0 0.0 0 0 ? S Feb02 0:00 [watchdog/1]
root 9 0.0 0.0 0 0 ? S Feb02 5:17 [events/0]
root 10 0.0 0.0 0 0 ? S Feb02 2:06 [events/1]
root 11 0.0 0.0 0 0 ? S Feb02 0:00 [cpuset]
root 12 0.0 0.0 0 0 ? S Feb02 0:00 [khelper]
root 13 0.0 0.0 0 0 ? S Feb02 0:00 [netns]
root 14 0.0 0.0 0 0 ? S Feb02 0:00 [async/mgr]
root 15 0.0 0.0 0 0 ? S Feb02 0:00 [pm]
root 17 0.0 0.0 0 0 ? S Feb02 0:45 [sync_supers]
root 18 0.0 0.0 0 0 ? S Feb02 0:47 [bdi-default]
root 19 0.0 0.0 0 0 ? S Feb02 0:00 [kintegrityd/0]
root 20 0.0 0.0 0 0 ? S Feb02 0:00 [kintegrityd/1]
root 21 0.0 0.0 0 0 ? S Feb02 0:47 [kblockd/0]
root 22 0.0 0.0 0 0 ? S Feb02 0:27 [kblockd/1]
root 23 0.0 0.0 0 0 ? S Feb02 0:00 [kacpid]
root 24 0.0 0.0 0 0 ? S Feb02 0:00 [kacpi_notify]
root 25 0.0 0.0 0 0 ? S Feb02 0:00 [kacpi_hotplug]
root 26 0.0 0.0 0 0 ? S Feb02 0:00 [ata_aux]
root 27 0.0 0.0 0 0 ? S Feb02 0:00 [ata_sff/0]
root 28 0.0 0.0 0 0 ? S Feb02 0:00 [ata_sff/1]
root 29 0.0 0.0 0 0 ? S Feb02 0:00 [khubd]
root 30 0.0 0.0 0 0 ? S Feb02 0:00 [kseriod]
root 31 0.0 0.0 0 0 ? S Feb02 0:00 [kmmcd]
root 32 0.0 0.0 0 0 ? S Feb02 0:07 [khungtaskd]
root 33 0.0 0.0 0 0 ? S Feb02 53:01 [kswapd0]
root 34 0.0 0.0 0 0 ? SN Feb02 0:00 [ksmd]
root 35 0.0 0.0 0 0 ? S Feb02 0:00 [aio/0]
root 36 0.0 0.0 0 0 ? S Feb02 0:00 [aio/1]
root 37 0.0 0.0 0 0 ? S Feb02 0:00 [ecryptfs-kthrea]
root 38 0.0 0.0 0 0 ? S Feb02 0:00 [crypto/0]
root 39 0.0 0.0 0 0 ? S Feb02 0:00 [crypto/1]
root 44 0.0 0.0 0 0 ? S Feb02 0:00 [pciehpd]
root 45 0.0 0.0 0 0 ? S Feb02 0:00 [scsi_eh_0]
root 46 0.0 0.0 0 0 ? S Feb02 0:00 [scsi_eh_1]
root 47 0.0 0.0 0 0 ? S Feb02 0:00 [kstriped]
root 49 0.0 0.0 0 0 ? S Feb02 0:00 [kmpathd/0]
root 50 0.0 0.0 0 0 ? S Feb02 0:00 [kmpathd/1]
root 51 0.0 0.0 0 0 ? S Feb02 0:00 [kmpath_handlerd]
root 52 0.0 0.0 0 0 ? S Feb02 0:00 [ksnapd]
root 53 0.0 0.0 0 0 ? S Feb02 0:00 [kondemand/0]
root 54 0.0 0.0 0 0 ? S Feb02 0:00 [kondemand/1]
root 55 0.0 0.0 0 0 ? S Feb02 0:00 [kconservative/0]
root 56 0.0 0.0 0 0 ? S Feb02 0:00 [kconservative/1]
root 246 0.0 0.0 0 0 ? S Feb02 2:58 [mpt_poll_0]
root 256 0.0 0.0 0 0 ? S Feb02 0:00 [mpt/0]
root 259 0.0 0.0 0 0 ? S Feb02 0:00 [scsi_eh_2]
root 278 0.0 0.0 0 0 ? S Feb02 0:08 [kdmflush]
root 282 0.0 0.0 0 0 ? S Feb02 0:00 [kdmflush]
root 297 0.0 0.0 0 0 ? S Feb02 0:36 [jbd2/dm-0-8]
root 298 0.0 0.0 0 0 ? S Feb02 0:00 [ext4-dio-unwrit]
root 299 0.0 0.0 0 0 ? S Feb02 0:00 [ext4-dio-unwrit]
root 338 0.0 0.0 17096 0 ? S Feb02 0:00 upstart-udev-bridge --daemon
root 340 0.0 0.0 147836 80 ? Ss Mar02 7:54 /usr/sbin/apache2 -k start
root 349 0.0 0.0 16880 0 ? S<s Feb02 0:00 udevd --daemon
root 554 0.0 0.0 0 0 ? S Feb02 0:00 [kpsmoused]
root 577 0.0 0.0 0 0 ? S Feb02 127:26 [vmmemctl]
root 657 0.0 0.0 0 0 ? S Feb02 126:36 [jbd2/sdb1-8]
root 658 0.0 0.0 0 0 ? S Feb02 0:00 [ext4-dio-unwrit]
root 659 0.0 0.0 0 0 ? S Feb02 0:00 [ext4-dio-unwrit]
root 788 0.0 0.0 6128 4 tty4 Ss+ Feb02 0:00 /sbin/getty -8 38400 tty4
root 791 0.0 0.0 6128 4 tty5 Ss+ Feb02 0:00 /sbin/getty -8 38400 tty5
root 793 0.0 0.0 49312 104 ? Ss Feb02 0:04 /usr/sbin/sshd
root 796 0.0 0.0 6128 4 tty2 Ss+ Feb02 0:00 /sbin/getty -8 38400 tty2
root 797 0.0 0.0 6128 4 tty3 Ss+ Feb02 0:00 /sbin/getty -8 38400 tty3
root 801 0.0 0.0 6128 4 tty6 Ss+ Feb02 0:00 /sbin/getty -8 38400 tty6
daemon 804 0.0 0.0 18932 0 ? Ss Feb02 0:02 atd
root 805 0.0 0.0 21128 156 ? Ss Feb02 0:47 cron
root 851 0.0 0.0 11360 304 ? Ss Feb02 15:18 /usr/sbin/irqbalance
root 1374 0.0 0.0 0 0 ? S Mar16 0:01 [jbd2/dm-2-8]
root 1375 0.0 0.0 0 0 ? S Mar16 0:00 [ext4-dio-unwrit]
root 1376 0.0 0.0 0 0 ? S Mar16 0:00 [ext4-dio-unwrit]
root 4088 0.0 0.0 0 0 ? S 11:58 0:00 [flush-251:0]
root 4329 0.0 0.0 70660 720 ? Ss 11:59 0:00 sshd: myuser [priv]
myuser 4347 0.0 0.0 70660 744 ? S 12:00 0:00 sshd: myuser@pts/1
myuser 4348 0.0 0.1 26028 6432 pts/1 Ss 12:00 0:00 -bash
munin 5150 0.0 0.0 39916 2040 ? Ss Apr08 8:04 /usr/sbin/munin-node
redis 8110 3.0 38.7 2244244 1553116 ? Ss May19 27:58 /usr/bin/redis-server /etc/redis/re
www-data 8433 0.0 0.0 148344 1928 ? S May17 0:01 /usr/sbin/apache2 -k start
www-data 8435 0.0 0.0 148336 2132 ? S May17 0:01 /usr/sbin/apache2 -k start
myuser 10115 0.0 0.0 26624 592 ? Ss 12:50 0:00 SCREEN -dR
myuser 10116 0.0 0.0 22808 2548 pts/0 Ss 12:50 0:00 /bin/bash
root 11410 0.0 0.0 27248 396 ? Ss Mar16 0:03 rpc.idmapd
root 11439 0.0 0.0 0 0 ? S Mar16 0:00 [lockd]
root 11440 0.0 0.0 0 0 ? S Mar16 0:03 [nfsd4]
root 11441 0.0 0.0 0 0 ? S Mar16 0:00 [nfsd4_callbacks]
root 11442 0.0 0.0 0 0 ? S Mar16 0:19 [nfsd]
root 11443 0.0 0.0 0 0 ? S Mar16 0:21 [nfsd]
root 11444 0.0 0.0 0 0 ? S Mar16 0:20 [nfsd]
root 11445 0.0 0.0 0 0 ? S Mar16 0:22 [nfsd]
root 11446 0.0 0.0 0 0 ? S Mar16 0:21 [nfsd]
root 11447 0.0 0.0 0 0 ? S Mar16 0:20 [nfsd]
root 11448 0.0 0.0 0 0 ? S Mar16 0:21 [nfsd]
root 11449 0.0 0.0 0 0 ? S Mar16 0:20 [nfsd]
root 11453 0.0 0.0 19032 316 ? Ss Mar16 0:09 /usr/sbin/rpc.mountd --manage-gids
myapp 11886 0.0 0.0 11984 196 pts/0 S+ 13:03 0:00 /bin/bash shared/worker.sh
myapp 11887 0.1 2.0 180724 81252 pts/0 S+ 13:04 0:03 resque-1.10.0: Forked 14745 at 1305
myapp 13085 0.1 2.0 195248 83180 ? S 13:17 0:03 Rails: /var/www/myapp-prd/current
myapp 14745 80.9 2.0 185908 80900 pts/0 R+ 13:29 19:09 resque-1.10.0: Processing myapp_import
www-data 15516 0.0 0.0 148224 1608 ? S May17 0:01 /usr/sbin/apache2 -k start
myuser 17579 0.0 0.0 17760 1220 pts/1 R+ 13:53 0:00 ps aux
mysql 18736 2.5 1.1 345568 46396 ? Ssl Mar16 2349:04 /usr/sbin/mysqld
daemon 21340 0.0 0.0 8304 4 ? Ss Mar16 0:00 portmap
statd 21564 0.0 0.0 14600 4 ? Ss Mar16 0:00 rpc.statd -L
root 21823 0.0 0.0 0 0 ? S Mar16 0:00 [rpciod/0]
root 21824 0.0 0.0 0 0 ? S Mar16 0:00 [rpciod/1]
www-data 22353 0.0 0.0 148484 1352 ? S May16 0:02 /usr/sbin/apache2 -k start
root 23471 0.0 0.0 0 0 ? S< Mar16 0:00 [kslowd000]
root 23472 0.0 0.0 0 0 ? S< Mar16 0:00 [kslowd001]
root 23473 0.0 0.0 0 0 ? S Mar16 0:00 [nfsiod]
www-data 27001 0.0 0.0 148496 1588 ? S May17 0:01 /usr/sbin/apache2 -k start
root 27365 0.0 0.0 23432 20 ? Ssl May15 0:00 PassengerWatchdog
root 27368 0.0 0.0 164064 1064 ? Sl May15 6:51 PassengerHelperAgent
root 27370 0.0 0.2 61904 8408 ? S May15 0:02 Passenger spawn server
nobody 27373 0.0 0.0 72172 28 ? Sl May15 0:01 PassengerLoggingAgent
www-data 27383 0.0 0.0 148356 1740 ? S May15 0:02 /usr/sbin/apache2 -k start
www-data 27384 0.0 0.0 148492 1444 ? S May15 0:02 /usr/sbin/apache2 -k start
www-data 27385 0.0 0.0 148224 1568 ? S May15 0:02 /usr/sbin/apache2 -k start
www-data 27386 0.0 0.0 148496 1860 ? S May15 0:02 /usr/sbin/apache2 -k start
root 29402 0.0 0.0 16876 4 ? S< Mar16 0:00 udevd --daemon
root 29800 0.0 0.0 0 0 ? S Mar16 0:00 [kdmflush]
root 29804 0.0 0.0 16876 4 ? S< Mar16 0:00 udevd --daemon
www-data 30231 0.0 0.0 148352 1904 ? S May16 0:02 /usr/sbin/apache2 -k start
root 31072 0.0 0.0 0 0 ? S May12 0:37 [flush-8:16]
syslog 31116 0.0 0.0 126028 388 ? Sl May12 0:04 rsyslogd -c4
root 32217 0.0 0.0 6128 4 tty1 Ss+ Mar17 0:00 /sbin/getty -8 38400 tty1
cat /proc/meminfo:
MemTotal: 4010060 kB
MemFree: 155604 kB
Buffers: 48232 kB
Cached: 149308 kB
SwapCached: 42036 kB
Active: 1572044 kB
Inactive: 639616 kB
Active(anon): 1496792 kB
Inactive(anon): 517464 kB
Active(file): 75252 kB
Inactive(file): 122152 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 741372 kB
SwapFree: 64 kB
Dirty: 1532 kB
Writeback: 0 kB
AnonPages: 1972572 kB
Mapped: 7920 kB
Shmem: 96 kB
Slab: 24384 kB
SReclaimable: 12224 kB
SUnreclaim: 12160 kB
KernelStack: 1368 kB
PageTables: 11632 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2746400 kB
Committed_AS: 3150752 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 281448 kB
VmallocChunk: 34359449320 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 12288 kB
DirectMap2M: 4132864 kB
One of the things I notice is the high value of Active(anon): 1496792 kB, which according to the docs is:
Memory that has been used more recently and usually not reclaimed unless absolutely necessary.
That's almost 1.5GB right there, is that normal? I do have very long (permanent) running processes (Resque) that fork each time a job comes in.
free, and ofps aux? Maybe we spot something.free,ps aux, andcat /proc/meminfooutput.freeoutput compared to the memory in use.