5

L2Arc is usually configured to cache random spinning platter reads on an SSD. I've set up this configuration in the hopes of speeding up directory traversal.

This is the setup:

# zpool list -v
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
geek1  2.72T  1.18T  1.54T         -    53%    43%  4.64x  ONLINE  -
  sdc  1.36T   606G   786G         -    54%    43%
  sdd  1.36T   606G   786G         -    53%    43%
log      -      -      -         -      -      -
  zil  3.97G      0  3.97G         -     0%     0%
cache      -      -      -         -      -      -
  l2arc  32.0G  4.24G  27.8G         -     0%    13%

The machine has 20GB of memory: plenty for Arc.

I would expect a find call over the filesystem to be sped up a lot. It is sped up by a factor of 5, but in absolute terms it's still slow. It takes 3.4 hours to list 46M files.

# while true; do time find /geek1/ -type f|wc -l; done
46774149

real    1103m18.661s
user    2m15.143s
sys     5m52.096s
46774149

real    243m0.289s
user    2m3.063s
sys     5m7.107s
46774149

real    205m23.922s
user    2m7.566s
sys     6m32.638s

How can directory traversal be made more efficient?

zpool get all prints:

NAME   PROPERTY                    VALUE                       SOURCE
geek1  size                        2.72T                       -
geek1  capacity                    43%                         -
geek1  altroot                     -                           default
geek1  health                      ONLINE                      -
geek1  guid                                                    default
geek1  version                     -                           default
geek1  bootfs                      -                           default
geek1  delegation                  on                          default
geek1  autoreplace                 off                         default
geek1  cachefile                   -                           default
geek1  failmode                    wait                        default
geek1  listsnapshots               off                         default
geek1  autoexpand                  off                         default
geek1  dedupditto                  0                           default
geek1  dedupratio                  4.64x                       -
geek1  free                        1.54T                       -
geek1  allocated                   1.18T                       -
geek1  readonly                    off                         -
geek1  ashift                      0                           default
geek1  comment                     -                           default
geek1  expandsize                  -                           -
geek1  freeing                     0                           default
geek1  fragmentation               53%                         -
geek1  leaked                      0                           default
geek1  feature@async_destroy       enabled                     local
geek1  feature@empty_bpobj         active                      local
geek1  feature@lz4_compress        active                      local
geek1  feature@spacemap_histogram  active                      local
geek1  feature@enabled_txg         active                      local
geek1  feature@hole_birth          active                      local
geek1  feature@extensible_dataset  enabled                     local
geek1  feature@embedded_data       active                      local
geek1  feature@bookmarks           enabled                     local
geek1  feature@filesystem_limits   enabled                     local
geek1  feature@large_blocks        enabled                     local
Improve this question
14
  • 1
    This is my archive and backup system. It archives projects I'm currently not working on. I frequently want to search in it to find files. I could add a locatedb, but figured adding an ssd cache would do the trick too, since on my ssd-only system 'find -type f' only takes a few seconds.'
    – Jos van den Oever
    May 27, 2017 at 16:38
  • 3
    WD Green Slow drives that have extremely poor seek times and very low IOPS/sec performance. Drives like this are probably just about the worst drives you can find for doing things like searching through directories for files. Note also that your ZIL is completely unused. ZIL is only used for synchronous writes. Normal user-space applications don't do synchronous writes. Things like well-designed databases do.
    – Andrew Henle
    May 28, 2017 at 13:20
  • 1
    That's why I've added an ssd cache for random reads. Writing is fast: backups perform well, but finding files again is really slow even with the Arc and L2Arc. So I'm wondering if they are configured properly. The L2Arc is not full, but traversing directories is still very slow.
    – Jos van den Oever
    May 28, 2017 at 13:41
  • 1
    Explanation from a former ZFS engineer - What you're doing doesn't make sense: reddit.com/r/zfs/comments/6doiqn/…
    – ewwhite
    Jun 7, 2017 at 12:06
  • 1
    poppycock. Of course L2ARC would help to improve performance if you'd compare it to pure rotational disks
    – poige
    Jun 8, 2017 at 10:10

1 Answer 1

Reset to default
1

If you are using ZFS on Linux, the slowness probably depends on xattr default setting (xattr=on), which stored extended attributes inside hidden directories/files.

Try setting chaning xattr to sa, issuing something like zfs set xattr=sa <pool/fs>. However, it affect only newly create files; so you need to remove/reload all files to get a significant speed up.

In other words, give it a small-scale try before reloading all your files.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .