6

We have two (or more sometimes) applications battling for the OS page cache. In particular ElasticSearch is very dependent on the OS page cache. To the point that cache misses can make or break performance in low latency cases. ES is the only low latency application on the nodes so it needs the OS page cache. Since the entire index can't fit into RAM warmers are used to prime the cache but after a few hours, those caches are trashed by other applications.

Some thoughts and ideas I've had:

  • RAMFS (but the index doesn't fit into RAM so no go there)
  • TMPFS (but it's not persisted through reboots and uses swap)
  • Caching at the Lucene/ES level (simply doesn't seem to exist, OS page cache is their caching strategy)
  • Dedicated hardware (this would be hard to justify since the machine have plenty of RAM and CPU, just need to make better use of it; plus we would need a lot of nodes).
  • VMTouch with locking (no way to really detect when ES is going to merge new segments and need to unlock the old files and map the new)
  • Full VM (performance cost and extra administration/management)

A dedicated file system that could carve out a predefined chunk of RAM for caching would be ideal. But I'm open to just about anything at this point short of buying dedicated nodes due to a design limitation in ES/Lucene.

closed as off-topic by John, Craig Watson, mdpc, Katherine Villyard, Hyppy Feb 6 '15 at 16:48

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    I don't have experience with ZFS but perhaps you should investigate its ARC page replacement algorithm and whether it might provide a benefit in your case. – HTTP500 Feb 5 '15 at 20:24
  • 1
    ZFS will probably help, but this use case really needs a dedicated VM for ElasticSearch with enough RAM to give acceptable performance. – Michael Hampton Feb 5 '15 at 20:41
  • How about moving competing applications off of this host onto a different one? – Matthew Ife Feb 5 '15 at 20:51
  • @MatthewIfe We are considering this as an option as well. Currently we enjoy the 1-to-1 ratio of our Hadoop and ES nodes. Makes for really fast updates/rebuilds of our index. As such we are exploring options to keep them located on the same machines. – KennyCason Feb 5 '15 at 20:55
  • (I work with @AndrewWhite) – KennyCason Feb 5 '15 at 21:14