6

We have two (or more sometimes) applications battling for the OS page cache. In particular ElasticSearch is very dependent on the OS page cache. To the point that cache misses can make or break performance in low latency cases. ES is the only low latency application on the nodes so it needs the OS page cache. Since the entire index can't fit into RAM warmers are used to prime the cache but after a few hours, those caches are trashed by other applications.

Some thoughts and ideas I've had:

  • RAMFS (but the index doesn't fit into RAM so no go there)
  • TMPFS (but it's not persisted through reboots and uses swap)
  • Caching at the Lucene/ES level (simply doesn't seem to exist, OS page cache is their caching strategy)
  • Dedicated hardware (this would be hard to justify since the machine have plenty of RAM and CPU, just need to make better use of it; plus we would need a lot of nodes).
  • VMTouch with locking (no way to really detect when ES is going to merge new segments and need to unlock the old files and map the new)
  • Full VM (performance cost and extra administration/management)

A dedicated file system that could carve out a predefined chunk of RAM for caching would be ideal. But I'm open to just about anything at this point short of buying dedicated nodes due to a design limitation in ES/Lucene.

7
  • 1
    I don't have experience with ZFS but perhaps you should investigate its ARC page replacement algorithm and whether it might provide a benefit in your case.
    – HTTP500
    Feb 5, 2015 at 20:24
  • 1
    ZFS will probably help, but this use case really needs a dedicated VM for ElasticSearch with enough RAM to give acceptable performance. Feb 5, 2015 at 20:41
  • How about moving competing applications off of this host onto a different one? Feb 5, 2015 at 20:51
  • @MatthewIfe We are considering this as an option as well. Currently we enjoy the 1-to-1 ratio of our Hadoop and ES nodes. Makes for really fast updates/rebuilds of our index. As such we are exploring options to keep them located on the same machines.
    – KennyCason
    Feb 5, 2015 at 20:55
  • (I work with @AndrewWhite)
    – KennyCason
    Feb 5, 2015 at 21:14

0

Browse other questions tagged .