Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

There is a index on Solr 4.5 which contains ~50K documents taking ~4Gb on disk. Physical RAM available to JVM is 6Gb (out of total 10Gb on server), OS is 64bit CentOS 5.

One of the applications queries that index several times in a loop, one request sent as soon as the previous one completes with virtually no delay in between but still not simultaneously.

The queries are all the same and fit the following simple pattern:

  • q=description:"keyword" OR orders:"keyword" OR catalog:"keyword"
  • fl=id,score,name,country
  • sort=score desc
  • wt=json
  • indent=true
  • start=0
  • rows=100

(all the fields in q are indexed tokenised text ones)

After a relatively small number of such queries (sometimes after the 3rd one, sometimes after the 5th etc.) received garbage collector on Solr side kicks in because all the memory is busy. That results in several seconds long delay before Solr responds which causes the application sending the request to abort on timeout.

If the application waits between the consequent queries in sends out for about 1 sec, all of them complete quickly and RAM isn't maxed out; but the quicker the queries are sent one after another, the greater are changes the next one would result in a GC triggered and an unwanted timeout.

The same index (built from the same data by the same application) is absolutely fine with the same usage pattern on the older version of Solr (1.4).

Increasing the cache sizes in solrconfig.xml doesn't seem to have much effect, but reducing rows parameter to 10 from 100 makes the timeout less likely, while not entirely impossible. queryResultWindowSize in the config is set to default 20.

I guess there might be more than one reason for the observed behaviour and I have given not enough details. If so, what do I need to change or profile to narrow it down from here?

share|improve this question

2 Answers 2

up vote 1 down vote accepted

This is the nature of Java: Garbage collection means the entire world stops.

The accepted solutions I know of are all generally lousy:

  1. Add more RAM (so you can do more queries before the memory pressure forces an automatic GC);
  2. Add more servers (to spread the load out) ; or
  3. Manually encourage the GC to run more frequently (so it's got less to do and completes faster)

Since you only started experiencing problems after upgrading to a new version of Solr you may want to check on their mailing lists and in their issue tracker to see if this is a known problem resolved in a later version.
If you want to profile and debug the application itself that would really be more of a Stack Overflow question -- it's possible you're Solr is something sub-optimal (in your query or in the app) that's causing it to use more RAM than it needs to.

share|improve this answer

Yuriy, I work at Azul Systems. We put our Zing virtual machine through its paces with Lucene in a benchmark test last year. The solution brief we authored shows representative latency stats in the graphic on page two: http://www.azulsystems.com/partners/apache-lucene-solr.

The Zing garbage collector "C4" is unlike those in Hotspot. Garbage collection under Zing means the entire world does not stop. Rather, Java mutator threads keep going, with GC operations running concurrently in parallel, typically on separate cores. There is no stop the world mode in Zing, either in primary GC tactics or as a fallback, in either the new and old generation collectors.

I cannot say we've seen Solr 4.5 as a particular culprit. If you are certain Solr or your app don't have a bug, and you still have GC pauses after triage, please consider looking at Zing: http://www.azulsystems.com/products/zing/whatisit

Voretaq7's comments on alternatives being lousy must not have included C4 or Zing. At least I hope so.

Good luck.

Matt

share|improve this answer
    
Hi Matt, this is interesting but changing the VM is a big change to our infrastructure, so I've been trying to tune the existing platform. –  Yuriy May 8 at 15:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.