We have two (or more sometimes) applications battling for the OS page cache. In particular ElasticSearch is very dependent on the OS page cache. To the point that cache misses can make or break performance in low latency cases. ES is the only low latency application on the nodes so it needs the OS page cache. Since the entire index can't fit into RAM warmers are used to prime the cache but after a few hours, those caches are trashed by other applications.
Some thoughts and ideas I've had:
- RAMFS (but the index doesn't fit into RAM so no go there)
- TMPFS (but it's not persisted through reboots and uses swap)
- Caching at the Lucene/ES level (simply doesn't seem to exist, OS page cache is their caching strategy)
- Dedicated hardware (this would be hard to justify since the machine have plenty of RAM and CPU, just need to make better use of it; plus we would need a lot of nodes).
- VMTouch with locking (no way to really detect when ES is going to merge new segments and need to unlock the old files and map the new)
- Full VM (performance cost and extra administration/management)
A dedicated file system that could carve out a predefined chunk of RAM for caching would be ideal. But I'm open to just about anything at this point short of buying dedicated nodes due to a design limitation in ES/Lucene.