2

We use Slicehost, with 512MB instances. We run Ubuntu 9.10 on them. I installed a few packages, and I'm now trying to optimize RAM consumption before running anything on there.

A simple ps gives me the list of running processes :

# ps faux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S<   Jan04   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S<   Jan04   0:15  \_ [migration/0]
root         4  0.0  0.0      0     0 ?        S<   Jan04   0:01  \_ [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [watchdog/0]
root         6  0.0  0.0      0     0 ?        S<   Jan04   0:04  \_ [events/0]
root         7  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [cpuset]
root         8  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [khelper]
root         9  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [async/mgr]
root        10  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xenwatch]
root        11  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xenbus]
root        13  0.0  0.0      0     0 ?        S<   Jan04   0:02  \_ [migration/1]
root        14  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [ksoftirqd/1]
root        15  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [watchdog/1]
root        16  0.0  0.0      0     0 ?        S<   Jan04   0:07  \_ [events/1]
root        17  0.0  0.0      0     0 ?        S<   Jan04   0:02  \_ [migration/2]
root        18  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [ksoftirqd/2]
root        19  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [watchdog/2]
root        20  0.0  0.0      0     0 ?        R<   Jan04   0:07  \_ [events/2]
root        21  0.0  0.0      0     0 ?        S<   Jan04   0:04  \_ [migration/3]
root        22  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [ksoftirqd/3]
root        23  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [watchdog/3]
root        24  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [events/3]
root        25  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kintegrityd/0]
root        26  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kintegrityd/1]
root        27  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kintegrityd/2]
root        28  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kintegrityd/3]
root        29  0.0  0.0      0     0 ?        S<   Jan04   0:01  \_ [kblockd/0]
root        30  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kblockd/1]
root        31  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kblockd/2]
root        32  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kblockd/3]
root        33  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kseriod]
root        34  0.0  0.0      0     0 ?        S    Jan04   0:00  \_ [khungtaskd]
root        35  0.0  0.0      0     0 ?        S    Jan04   0:05  \_ [pdflush]
root        36  0.0  0.0      0     0 ?        S    Jan04   0:06  \_ [pdflush]
root        37  0.0  0.0      0     0 ?        S<   Jan04   1:02  \_ [kswapd0]
root        38  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [aio/0]
root        39  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [aio/1]
root        40  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [aio/2]
root        41  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [aio/3]
root        42  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [jfsIO]
root        43  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [jfsCommit]
root        44  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [jfsCommit]
root        45  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [jfsCommit]
root        46  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [jfsCommit]
root        47  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [jfsSync]
root        48  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfs_mru_cache]
root        49  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfslogd/0]
root        50  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfslogd/1]
root        51  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfslogd/2]
root        52  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfslogd/3]
root        53  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfsdatad/0]
root        54  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfsdatad/1]
root        55  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfsdatad/2]
root        56  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfsdatad/3]
root        57  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfsconvertd/0]
root        58  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfsconvertd/1]
root        59  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfsconvertd/2]
root        60  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [xfsconvertd/3]
root        61  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [glock_workqueue]
root        62  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [glock_workqueue]
root        63  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [glock_workqueue]
root        64  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [glock_workqueue]
root        65  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [delete_workqueu]
root        66  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [delete_workqueu]
root        67  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [delete_workqueu]
root        68  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [delete_workqueu]
root        69  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kslowd]
root        70  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kslowd]
root        71  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [crypto/0]
root        72  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [crypto/1]
root        73  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [crypto/2]
root        74  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [crypto/3]
root        77  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [net_accel/0]
root        78  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [net_accel/1]
root        79  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [net_accel/2]
root        80  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [net_accel/3]
root        81  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [sfc_netfront/0]
root        82  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [sfc_netfront/1]
root        83  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [sfc_netfront/2]
root        84  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [sfc_netfront/3]
root       310  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [kstriped]
root       315  0.0  0.0      0     0 ?        S<   Jan04   0:00  \_ [ksnapd]
root      1452  0.0  0.0      0     0 ?        S<   Jan04   4:31  \_ [kjournald]
root         1  0.0  0.1  19292   948 ?        Ss   Jan04   0:15 /sbin/init
root      1545  0.0  0.1  13164  1064 ?        S    Jan04   0:00 upstart-udev-bridge --daemon
root      1547  0.0  0.1  17196   996 ?        S<s  Jan04   0:00 udevd --daemon
root      1728  0.0  0.2  20284  1468 ?        S<   Jan04   0:00  \_ udevd --daemon
root      1729  0.0  0.1  17192   792 ?        S<   Jan04   0:00  \_ udevd --daemon
root      1881  0.0  0.0   8192   152 ?        Ss   Jan04   0:00 dd bs=1 if=/proc/kmsg of=/var/run/rsyslog/kmsg
syslog    1884  0.0  0.2 185252  1200 ?        Sl   Jan04   1:00 rsyslogd -c4
103       1894  0.0  0.1  23328   700 ?        Ss   Jan04   1:08 dbus-daemon --system --fork
root      2046  0.0  0.0    136    32 ?        Ss   Jan04   4:05 runsvdir -P /etc/service log: gems/custom_require.rb:31:in `require'??from /mnt/app/superfeedr-firehoser/current/script/component:52?/opt/ruby-enterprise/lib/ruby/si
root      2055  0.0  0.0    112    32 ?        Ss   Jan04   0:00  \_ runsv chef-client
root      2060  0.0  0.0    132    40 ?        S    Jan04   0:02  |   \_ svlogd -tt ./main
root      2056  0.0  0.0    112    28 ?        Ss   Jan04   0:20  \_ runsv superfeedr-firehoser_2
root      2059  0.0  0.0    132    40 ?        S    Jan04   0:29  |   \_ svlogd /var/log/superfeedr-firehoser_2
root      2057  0.0  0.0    112    28 ?        Ss   Jan04   0:20  \_ runsv superfeedr-firehoser_1
root      2062  0.0  0.0    132    44 ?        S    Jan04   0:26      \_ svlogd /var/log/superfeedr-firehoser_1
root      2058  0.0  0.0  18708   316 ?        Ss   Jan04   0:01 cron
root      2095  0.0  0.1  49072   764 ?        Ss   Jan04   0:06 /usr/sbin/sshd
root      9832  0.0  0.5  78916  3500 ?        Ss   00:37   0:00  \_ sshd: root@pts/0 
root      9846  0.0  0.3  17900  2036 pts/0    Ss   00:37   0:00      \_ -bash
root     10132  0.0  0.1  15020  1064 pts/0    R+   09:51   0:00          \_ ps faux
root      2180  0.0  0.0   5988   140 tty1     Ss+  Jan04   0:00 /sbin/getty -8 38400 tty1
root     27610  0.0  1.4  47060  8436 ?        S    Apr04   2:21 python /usr/sbin/denyhosts --daemon --purge --config=/etc/denyhosts.conf --config=/etc/denyhosts.conf
root     22640  0.0  0.7 119244  4164 ?        Ssl  Apr05   0:05 /usr/sbin/console-kit-daemon
root     10113  0.0  0.0   3904   316 ?        Ss   09:46   0:00 /usr/sbin/collectdmon -P /var/run/collectdmon.pid -- -C /etc/collectd/collectd.conf
root     10114  0.0  0.2 201084  1464 ?        Sl   09:46   0:00  \_ collectd -C /etc/collectd/collectd.conf -f

As you can see there is nothing serious here. If I sum up the RSS line on all this, I get the following :

#  ps -aeo rss | awk '{sum+=$1} END {print sum}'
30096

Which makes sense.

However, I have a pretty big surprise when I do a free:

# free
             total       used       free     shared    buffers     cached
Mem:        591180     343684     247496          0      25432     161256
-/+ buffers/cache:     156996     434184
Swap:      1048568          0    1048568

As you can see 60% of the available memory is already consumed... which leaves me with only 40% to run my own applications if I want to avoid swapping. Quite disapointing!

2 questions arise :

  • Where is all this memory?
  • How to take some of it back for my own apps?

3 Answers 3

13

You've got a total of 156996 KB of memory used for programs, and a further 343684-156996=186688 KB used for buffers & cache. That means that about 22% of memory is used, not 60%.

These buffers are used to store the contents of your disks, either before they're written to the disk, or after they've been read in case you want to read the data again. You can see this if you do an ls on a big directory. The first time it takes a significant amount of time, then the second time it's almost instant.

Linux allocates these buffer/cache memories on its own. However, if your programs expand, then the amount used for buffer/cache is likely to decrease. You really don't want to change the amount used, Linux's built in algorithms are much better at deciding the amount than you will be.

Also, you can't just add up the sizes of processes like you did. More than one process can use the same blocks of memory, for example through shared memory or fork(2), or simply by being the same executable program.

2

The used ram is simply down to Linux caching the file system. By default Linux is pretty aggressive with this. If applications require the RAM, Linux will evict some file system cache. If you find the system is swapping when your app starts requesting more RAM, can you adjust /proc/sys/vm/swapiness, but I can't imagine this would be necessary.

2

If you really want to have less RAM used by applications then you can put a higher value in /proc/sys/vm/swappiness in order to tell the kernel to force more stuff into swap when it's not being used. This may impact initial performance of less-used services though, since it will have to be loaded from swap.

echo -n "95" > /proc/sys/vm/swappiness

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .