20

I'm running ZoL 0.6.2 from their PPA on Ubuntu 12.04. It's on a host with 16GB of memory intended to run some VMs using KVM/Libvirt. After some time ZoL is using an insane amount of memory, reaching 98% of RAM usage with some VMs running. This results new processes refusing to start "unable to allocate memory". I can't even start all my VMs anymore which before using ZFS were using about 40-50% of RAM.

As far as I understand, without tweaking, ZoL should release memory as soon as the system is short on memory. Well, it doesn't. So I decided to set the arc_max setting to 1GB.

# echo 1073741824 >> /sys/module/zfs/parameters/zfs_arc_max

Still, it does not release any memory.

As you can see from the below ARC statistics, it's using more memory than it's configured to (compare c=7572030912 with c_max=1073741824).

What am I doing wrong here?

# cat /proc/spl/kstat/zfs/arcstats
4 1 0x01 84 4032 43757119584 392054268420115
name                            type data
hits                            4    28057644
misses                          4    13975282
demand_data_hits                4    19632274
demand_data_misses              4    571809
demand_metadata_hits            4    6333604
demand_metadata_misses          4    289110
prefetch_data_hits              4    1903379
prefetch_data_misses            4    12884520
prefetch_metadata_hits          4    188387
prefetch_metadata_misses        4    229843
mru_hits                        4    15390332
mru_ghost_hits                  4    1088944
mfu_hits                        4    10586761
mfu_ghost_hits                  4    169152
deleted                         4    35432344
recycle_miss                    4    701686
mutex_miss                      4    35304
evict_skip                      4    60416647
evict_l2_cached                 4    0
evict_l2_eligible               4    3022396862976
evict_l2_ineligible             4    1602907651584
hash_elements                   4    212777
hash_elements_max               4    256438
hash_collisions                 4    17163377
hash_chains                     4    51485
hash_chain_max                  4    10
p                               4    1527347963
c                               4    7572030912
c_min                           4    1038188800
c_max                           4    1073741824
size                            4    7572198224
hdr_size                        4    66873056
data_size                       4    7496095744
other_size                      4    9229424
anon_size                       4    169150464
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    1358216192
mru_evict_data                  4    1352400896
mru_evict_metadata              4    508928
mru_ghost_size                  4    6305992192
mru_ghost_evict_data            4    4919159808
mru_ghost_evict_metadata        4    1386832384
mfu_size                        4    5968729088
mfu_evict_data                  4    5627991552
mfu_evict_metadata              4    336846336
mfu_ghost_size                  4    1330455552
mfu_ghost_evict_data            4    1287782400
mfu_ghost_evict_metadata        4    42673152
l2_hits                         4    0
l2_misses                       4    0
l2_feeds                        4    0
l2_rw_clash                     4    0
l2_read_bytes                   4    0
l2_write_bytes                  4    0
l2_writes_sent                  4    0
l2_writes_done                  4    0
l2_writes_error                 4    0
l2_writes_hdr_miss              4    0
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    0
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    0
l2_asize                        4    0
l2_hdr_size                     4    0
l2_compress_successes           4    0
l2_compress_zeros               4    0
l2_compress_failures            4    0
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    66583
memory_indirect_count           4    7657293
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    427048272
arc_meta_limit                  4    2076377600
arc_meta_max                    4    498721632

# free -m
             total       used       free     shared    buffers     cached
Mem:         15841      15385        456          0         75         74
-/+ buffers/cache:      15235        606
Swap:            0          0          0
22

IMHO the parameters in /sys/module/zfs/parameters can only be set to 0 / 1 - disabled / enabled." Correction: depends on the parameter

I am in the same boat wanting to limit zfs' memory usage and it seems one has to create a /etc/modprobe.d/zfs.conf file and enter the parameter and the desired value in there. This change will take effect upon reboot.

echo "options zfs zfs_arc_max=34359738368" >> /etc/modprobe.d/zfs.conf

To effect the running module one can change the zfs_arc_max parameter.

echo "34359738368" > /sys/module/zfs/parameters/zfs_arc_max

Please note the use of > to replace the content of the file in contrast to adding to the file with >>.

source: https://stackoverflow.com/a/18808311

  • 1
    The ZFS arc does not shrink immediately. However (ZFSonLinux) it is reclaimed while applications allocate that memory - as usual. If you need something to hog on memory, perhaps have a look at github.com/hilbix/killmem (only 8K after make static; strip -s killmem) – Tino Aug 24 '15 at 9:12
  • On Ubuntu 16.04 I needed to run update-initramfs -u -k all before reboot to have this settings from /etc/modprobe.d/zfs.conf propagated. – lechup Jul 15 '17 at 9:13
  • @lechup: On Ubuntu 16.04, I added options zfs zfs_vdev_scheduler=cfq to my /etc/modprobe.d/zfs.conf. I rebooted and it worked; the scheduler was now cfq instead of noop. Can you elaborate why update-initramfs -u -k all is necessary? – Martin Velez Feb 2 '18 at 8:28
  • @MartinVelez I know it's weird but without it on my machine changes weren't propagated after reboot... I was trying to popagate different option zfs_arc_max maybe this key is somehow cached in initramfs? – lechup Feb 2 '18 at 13:19
5

http://fibrevillage.com/storage/169-zfs-arc-on-linux-how-to-set-and-monitor-on-linux

this article is very good

starting version ZoL 0.6.2 you can set the c_max at runtime, however the ARC size will not be released automatically. to force the RAM to be released it is needed to export the zpool.

4

Once you modify your arc size, you need to drop your caches.

echo 3 > /proc/sys/vm/drop_caches

and wait (your prompt won't return immediately, but other processes will continue to run). It will slowly unload the caches (2.5 min for my 24GB cache on 2 pair of raid 1'd 2TB WD blacks on a 2Ghz 4 year old cpu on a box w/64GB) - beware, you will suddenly have no caches, and any processes reading data will be pulling off raw disk so you'll probably see IO wait jump up for a while til cache is repopulated.

  • Ah cool! Could you explain why the '3' as a value to write to that procfs entry? – gertvdijk Feb 18 '17 at 17:32
  • Clear PageCache only: # sync; echo 1 > /proc/sys/vm/drop_caches Clear dentries and inodes: # sync; echo 2 > /proc/sys/vm/drop_caches Clear PageCache, dentries and inodes: # sync; echo 3 > /proc/sys/vm/drop_caches – math Feb 19 '17 at 0:50
2

One problem you could be falling into is ZFS caching the virtual machine files (virtual disks). To avoid that I always set the primarycache property to "metadata" on filesystems containing virtual disks.

The logic being that the guest OS have better hint on what areas of its disks to cache.

0

AFAIK one of the following conditions must be met to adapt the parameter.

  1. On a running system: export all zpools, remove zfs modules, re-enable zfs module (per definition this can't be done if / is on zfs).
  2. Regenerate the initramfs image when changing the parameter so after a reboot it will work. This is needed as the zfs.conf file location is not yet mounted at that moment in the boot process.
0

You have one extra ">" too many.

The command should be

echo 1073741824 > /sys/module/zfs/parameters/zfs_arc_max

not ">>"

>> means "add to" (existing list).
> means "overwrite" (value).

This is why the command code in your question will not work.

  • That was already part of the accepted answer present. serverfault.com/a/602457/135437 – gertvdijk Jul 26 '18 at 21:40
  • That post is a convoluted mess, Mr. Downvotes. The author says a whole rack of things, only touching the correct answer in passing near the end of the mess, without stating, "this is the reason" or equivalent. So convoluted, that I didn't get my answer from that post. I saw the reason from the question. My post was upvoted for a reason. – Hypocritus Aug 23 '18 at 0:40

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.