24

I'm running ZoL 0.6.2 from their PPA on Ubuntu 12.04. It's on a host with 16GB of memory intended to run some VMs using KVM/Libvirt. After some time ZoL is using an insane amount of memory, reaching 98% of RAM usage with some VMs running. This results new processes refusing to start "unable to allocate memory". I can't even start all my VMs anymore which before using ZFS were using about 40-50% of RAM.

As far as I understand, without tweaking, ZoL should release memory as soon as the system is short on memory. Well, it doesn't. So I decided to set the arc_max setting to 1GB.

# echo 1073741824 >> /sys/module/zfs/parameters/zfs_arc_max

Still, it does not release any memory.

As you can see from the below ARC statistics, it's using more memory than it's configured to (compare c=7572030912 with c_max=1073741824).

What am I doing wrong here?

# cat /proc/spl/kstat/zfs/arcstats
4 1 0x01 84 4032 43757119584 392054268420115
name                            type data
hits                            4    28057644
misses                          4    13975282
demand_data_hits                4    19632274
demand_data_misses              4    571809
demand_metadata_hits            4    6333604
demand_metadata_misses          4    289110
prefetch_data_hits              4    1903379
prefetch_data_misses            4    12884520
prefetch_metadata_hits          4    188387
prefetch_metadata_misses        4    229843
mru_hits                        4    15390332
mru_ghost_hits                  4    1088944
mfu_hits                        4    10586761
mfu_ghost_hits                  4    169152
deleted                         4    35432344
recycle_miss                    4    701686
mutex_miss                      4    35304
evict_skip                      4    60416647
evict_l2_cached                 4    0
evict_l2_eligible               4    3022396862976
evict_l2_ineligible             4    1602907651584
hash_elements                   4    212777
hash_elements_max               4    256438
hash_collisions                 4    17163377
hash_chains                     4    51485
hash_chain_max                  4    10
p                               4    1527347963
c                               4    7572030912
c_min                           4    1038188800
c_max                           4    1073741824
size                            4    7572198224
hdr_size                        4    66873056
data_size                       4    7496095744
other_size                      4    9229424
anon_size                       4    169150464
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    1358216192
mru_evict_data                  4    1352400896
mru_evict_metadata              4    508928
mru_ghost_size                  4    6305992192
mru_ghost_evict_data            4    4919159808
mru_ghost_evict_metadata        4    1386832384
mfu_size                        4    5968729088
mfu_evict_data                  4    5627991552
mfu_evict_metadata              4    336846336
mfu_ghost_size                  4    1330455552
mfu_ghost_evict_data            4    1287782400
mfu_ghost_evict_metadata        4    42673152
l2_hits                         4    0
l2_misses                       4    0
l2_feeds                        4    0
l2_rw_clash                     4    0
l2_read_bytes                   4    0
l2_write_bytes                  4    0
l2_writes_sent                  4    0
l2_writes_done                  4    0
l2_writes_error                 4    0
l2_writes_hdr_miss              4    0
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    0
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    0
l2_asize                        4    0
l2_hdr_size                     4    0
l2_compress_successes           4    0
l2_compress_zeros               4    0
l2_compress_failures            4    0
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    66583
memory_indirect_count           4    7657293
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    427048272
arc_meta_limit                  4    2076377600
arc_meta_max                    4    498721632

# free -m
             total       used       free     shared    buffers     cached
Mem:         15841      15385        456          0         75         74
-/+ buffers/cache:      15235        606
Swap:            0          0          0

7 Answers 7

32

IMHO the parameters in /sys/module/zfs/parameters can only be set to 0 / 1 - disabled / enabled." Correction: depends on the parameter

I am in the same boat wanting to limit zfs' memory usage and it seems one has to create a /etc/modprobe.d/zfs.conf file and enter the parameter and the desired value in there. This change will take effect upon reboot.

echo "options zfs zfs_arc_max=34359738368" >> /etc/modprobe.d/zfs.conf

To effect the running module one can change the zfs_arc_max parameter.

echo "34359738368" > /sys/module/zfs/parameters/zfs_arc_max

Please note the use of > to replace the content of the file in contrast to adding to the file with >>.

source: https://stackoverflow.com/a/18808311

5
  • 1
    The ZFS arc does not shrink immediately. However (ZFSonLinux) it is reclaimed while applications allocate that memory - as usual. If you need something to hog on memory, perhaps have a look at github.com/hilbix/killmem (only 8K after make static; strip -s killmem)
    – Tino
    Aug 24, 2015 at 9:12
  • 2
    On Ubuntu 16.04 I needed to run update-initramfs -u -k all before reboot to have this settings from /etc/modprobe.d/zfs.conf propagated.
    – lechup
    Jul 15, 2017 at 9:13
  • 1
    @lechup: On Ubuntu 16.04, I added options zfs zfs_vdev_scheduler=cfq to my /etc/modprobe.d/zfs.conf. I rebooted and it worked; the scheduler was now cfq instead of noop. Can you elaborate why update-initramfs -u -k all is necessary? Feb 2, 2018 at 8:28
  • @MartinVelez I know it's weird but without it on my machine changes weren't propagated after reboot... I was trying to popagate different option zfs_arc_max maybe this key is somehow cached in initramfs?
    – lechup
    Feb 2, 2018 at 13:19
  • I also had to run update-initramfs -u -k all on Ubuntu 22.04 to get options zfs zfs_dirty_data_max_max=... picked up from /etc/modprobe.d/zfs.conf Sep 8, 2022 at 4:56
13

Once you modify your arc size, you need to drop your caches.

echo 3 > /proc/sys/vm/drop_caches

and wait (your prompt won't return immediately, but other processes will continue to run). It will slowly unload the caches (2.5 min for my 24GB cache on 2 pair of raid 1'd 2TB WD blacks on a 2Ghz 4 year old cpu on a box w/64GB) - beware, you will suddenly have no caches, and any processes reading data will be pulling off raw disk so you'll probably see IO wait jump up for a while til cache is repopulated.

2
  • Ah cool! Could you explain why the '3' as a value to write to that procfs entry?
    – gertvdijk
    Feb 18, 2017 at 17:32
  • 1
    Clear PageCache only: # sync; echo 1 > /proc/sys/vm/drop_caches Clear dentries and inodes: # sync; echo 2 > /proc/sys/vm/drop_caches Clear PageCache, dentries and inodes: # sync; echo 3 > /proc/sys/vm/drop_caches
    – math
    Feb 19, 2017 at 0:50
4

http://fibrevillage.com/storage/169-zfs-arc-on-linux-how-to-set-and-monitor-on-linux

this article is very good

starting version ZoL 0.6.2 you can set the c_max at runtime, however the ARC size will not be released automatically. to force the RAM to be released it is needed to export the zpool.

2
  • 1
    Link broken Jan 2021
    – Fredrik
    Jan 18, 2021 at 7:50
  • you can drop caches instead of exporting the pool. see comment above.
    – math
    Oct 12, 2021 at 22:46
4

You have one extra ">" too many.

The command should be

echo 1073741824 > /sys/module/zfs/parameters/zfs_arc_max

not ">>"

>> means "add to" (existing list).
> means "overwrite" (value).

This is why the command code in your question will not work.

3
  • That was already part of the accepted answer present. serverfault.com/a/602457/135437
    – gertvdijk
    Jul 26, 2018 at 21:40
  • 3
    That post is a convoluted mess, Mr. Downvotes. The author says a whole rack of things, only touching the correct answer in passing near the end of the mess, without stating, "this is the reason" or equivalent. So convoluted, that I didn't get my answer from that post. I saw the reason from the question. My post was upvoted for a reason.
    – Hypocritus
    Aug 23, 2018 at 0:40
  • Unfortunately, the difference between > and >> (i.e. O_WRONLY | O_TRUNC vs O_APPEND) only holds for regular files in actual storage. The "files" in /proc and /sys are kernel interfaces, so reading/writing stuff have a different semantic.
    – iBug
    Jan 5, 2021 at 16:54
2

One problem you could be falling into is ZFS caching the virtual machine files (virtual disks). To avoid that I always set the primarycache property to "metadata" on filesystems containing virtual disks.

The logic being that the guest OS have better hint on what areas of its disks to cache.

0

AFAIK one of the following conditions must be met to adapt the parameter.

  1. On a running system: export all zpools, remove zfs modules, re-enable zfs module (per definition this can't be done if / is on zfs).
  2. Regenerate the initramfs image when changing the parameter so after a reboot it will work. This is needed as the zfs.conf file location is not yet mounted at that moment in the boot process.
0

Settings are not set when arc_max is lower than arc_min, you need to set arc_min too.

/etc/modprobe.d/zfs.conf
options zfs zfs_arc_min=0
options zfs zfs_arc_max=2147483648

you need rebuild initramfs with update-initramfs -u and reboot system

you can check

# arc_summary
ARC size (current):                                    10.6 %  216.9 MiB
    Target size (adaptive):                        97.9 %    2.0 GiB
    Min size (hard limit):                         97.9 %    2.0 GiB
    Max size (high water):                            1:1    2.0 GiB

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .