Are there any good guides for hardware level to support ElasticSearch? Are recommendations for Lucene or Solr a good place to start? We're looking at rolling out a deployment starting with

  • 27 million documents, 8TB of data
  • add 300k documents per day

Then scaling that up about 10x, to

  • 270 million documents, 80TB of data
  • add 3 million documents/day

This is a strange use case, where queries would be in the thousands/day, but response times need to remain low enough for a good experience with an Ajaxy webapp.

closed as too localized by Mark Henderson Jan 16 '12 at 3:38

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center. If this question can be reworded to fit the rules in the help center, please edit the question.

  • @MarkHenderson : this is a real (non-toy) and interesting question. I think your assessment of it being "too localized" is off-target. – David J. Nov 13 '12 at 15:06
  • David, the question was closed as per our FAQ we don't do shopping questions – Mark Henderson Nov 13 '12 at 19:05

There are a lot of factors that can come into play, so I don't think there are many general guidelines.

You should conduct a smaller scale evaluation, perhaps with 1/5th the initial data set to see how things behave when you throw your expected indexing and search load at the setup. This will ensure you understand how much space your data will actually consume in the search engine. For elasticsearch, depends if you are storing the source json and how fields get analyzed and if they are stored.

EC2 can be a reasonable way to eval elasticsearch without a large h/w expenditure.

For cluster based software, like elasticsearch, there are tradeoffs between keeping the cluster smaller vs larger. A large cluster is nice because when you lose a server, less data needs to be re-allocated. A smaller cluster consumes less energy and is easier to maintain.

We run a cluster with 35 million documents w/ total index size around 300GB x 2, since all the indexes are replicated. To support this and a very large number of searches, we have 4 nodes, each with 24 cores, 48GB of RAM and 1TB of storage with 10K disks in raid10. We recently increased disk size to ensure we had more head room.

For your case, I'd recommend more RAM and more disk. You can probably save money on CPUs with that search volume.

Low search volume actually hurts performance, since caches (both internal to the s/w used and OS disk) won't be warmed well.

Hope this helps, Paul

  • What kind of documents are you talking about? Logs? Real documents? – Manuel Rauber Aug 26 '13 at 13:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.