Regular Error 500 with SOLR-Server

Question

In the recent weeks the SOLR server system of our customers have the same error regularly, increasingly often. By now it's almost daily. In the client, the SOLR 3.6 throws this exception to the PHP Client:

java.lang.RuntimeException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/solr/data/spellchecker/write.lock
    at org.apache.solr.spelling.IndexBasedSpellChecker.build(IndexBasedSpellChecker.java:92)
    at org.apache.solr.handler.component.SpellCheckComponent.prepare(SpellCheckComponent.java:109)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:165)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/solr/data/spellchecker/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:84)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
    at org.apache.lucene.search.spell.SpellChecker.clearIndex(SpellChecker.java:535)
    at org.apache.solr.spelling.IndexBasedSpellChecker.build(IndexBasedSpellChecker.java:88)
    ... 18 more

This can only resolved restarting the Servlet-Container (Tomcat 6). However, talking to the SOLR over curl gives no trouble at all. Only the PHP-Client throws this error continuously. And as far as we know, we don't even touch any spellchecker module of SOLR at all. And the crashes seem to happen at times when there's no indexing happening.

I have tried to solve this by increasing the amount of RAM and the WriteLockTimeout to 20 seconds, but the problem persists.

Can anyone help with a hint on what is the problem?

Answer 1

It turns out it was a problem with the memory. As the tomcat was installed, someone left the max heap size of the JVM 128M, which caused the JVM to just stop allocating any memory. Instead of proper OutOfMemoryExceptions, we got these cryptic messages.

Answer 2

If you don't touch the spellchecker module at all, can you uncomment that section from your config and restart the service?

I think you're getting multiple "&spellcheck.build=true" requests (so it's trying to rebuild while it's already building, hence the lock) - if you log all your solr requests somewhere check for that and see if you can stop them?