How to fix Solr - Server is shutting down issue?

Question

I was having a running Solr 4.1 on Windows Server 2008 R2. The Solr is deployed on Tomcat. However, today it stops suddenly, and while accessing Solr it gives following error.

HTTP Status 503 - Server is shutting down

type Status report

message Server is shutting down

description The requested service is not currently available.

On further looking into Logs, we got following:

Log File: tomcat7-stderr.2013-05-09.txt

    May 09, 2013 8:00:40 PM org.apache.solr.core.CoreContainer finalize
    SEVERE: CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=2221663

Log File: catalina.2013-05-09.txt

May 09, 2013 7:59:25 PM org.apache.solr.core.SolrResourceLoader <init>
INFO: new SolrResourceLoader for directory: 'c:\solrdir\'
May 09, 2013 7:59:29 PM org.apache.solr.common.SolrException log
SEVERE: Exception during parsing file: null:org.xml.sax.SAXParseException; systemId: file:/c:/solr/solr.xml; lineNumber: 2; columnNumber: 6; The processing instruction target matching "[xX][mM][lL]" is not allowed.
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.scanPIData(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanPIData(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.scanPI(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
    at org.apache.solr.core.Config.<init>(Config.java:121)
    at org.apache.solr.core.CoreContainer.load(CoreContainer.java:428)
    at org.apache.solr.core.CoreContainer.load(CoreContainer.java:404)
    at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:336)
    at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:98)
    at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:281)
    at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:262)
    at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:107)
    at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4656)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5309)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:977)
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1655)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

May 09, 2013 7:59:29 PM org.apache.solr.servlet.SolrDispatchFilter init
SEVERE: Could not start Solr. Check solr/home property and the logs
May 09, 2013 7:59:29 PM org.apache.solr.common.SolrException log
SEVERE: null:org.apache.solr.common.SolrException: 
    at org.apache.solr.core.CoreContainer.load(CoreContainer.java:431)
    at org.apache.solr.core.CoreContainer.load(CoreContainer.java:404)
    at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:336)
    at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:98)
    at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:281)
    at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:262)
    at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:107)
    at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4656)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5309)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:977)
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1655)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: org.xml.sax.SAXParseException; systemId: file:/c:/solrdir/solr.xml; lineNumber: 2; columnNumber: 6; The processing instruction target matching "[xX][mM][lL]" is not allowed.
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.scanPIData(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanPIData(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.scanPI(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
    at org.apache.solr.core.Config.<init>(Config.java:121)
    at org.apache.solr.core.CoreContainer.load(CoreContainer.java:428)
    ... 20 more

May 09, 2013 7:59:29 PM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init() done
May 09, 2013 7:59:29 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\Program Files (x86)\Apache Software Foundation\Tomcat 7.0\webapps\docs
May 09, 2013 7:59:30 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\Program Files (x86)\Apache Software Foundation\Tomcat 7.0\webapps\manager
May 09, 2013 7:59:30 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\Program Files (x86)\Apache Software Foundation\Tomcat 7.0\webapps\ROOT
May 09, 2013 7:59:30 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8983"]
May 09, 2013 7:59:30 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
May 09, 2013 7:59:30 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 9578 ms
May 09, 2013 8:00:40 PM org.apache.solr.core.CoreContainer finalize
SEVERE: CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=2221663

Any idea what could be wrong and how to fix?

Krunal · Accepted Answer · 2013-05-09 15:05:50Z

2

I got it fixed by removing a blank line at the top of solr.xml file. But now a different error appear, although Solr seems to be running now.

This helped me to figure out the issue:

http://www.bennadel.com/blog/58-The-Processing-Instruction-Target-Matching-xX-mM-lL-is-Not-Allowed.htm

answered May 9, 2013 at 15:05

Krunal

2311 gold badge5 silver badges18 bronze badges

Add a comment |

Magus · Accepted Answer · 2014-05-27 17:46:50Z

1

I got the same problem when I was copying a core.

O copied the folder, in my case, users to qa.

When I changed the name=users to name=qa inside core.properties file, it all went back to work.

Every tiny bit matters, and as Solr was, sadly, written in Java the error messages are usually worthless.

To be safe about your XML, use something like http://www.xmlvalidation.com/ to make sure that it's not a simple syntax error ruining your day.

answered May 27, 2014 at 17:46

Magus

1112 bronze badges

Add a comment |

Michael Hampton · Accepted Answer · 2013-06-04 06:22:38Z

0

try closing the core tag in solr.xml; like:

<core name="eg1" instanceDir="eg1Dir"/>

edited Jun 4, 2013 at 6:22

Michael Hampton

248k45 gold badges512 silver badges978 bronze badges

answered Jun 4, 2013 at 6:10

hshah

11 bronze badge

Add a comment |

mydoghasworms · Accepted Answer · 2014-04-03 10:50:22Z

I eventually copied all the libraries under dist/solrj-lib (I am running on Linux) to /usr/share/tomcat7/lib. That got rid of the 503 error, but in addition I had to change the permissions on the $SOLR_HOME directory and sub-items to be owned by user and group tomcat7 and I had to edit the solr.xml file so I was left with only <solr/>.

It's quite a mission to set up Solr these days. It would be nice if they could just provide a sane set of defaults and include libraries to get everything up and running without all these steps.

theMarceloR · Accepted Answer · 2014-12-01 18:23:01Z

Just to document this alternative solution in case someone else is facing this issue (however, without seeing the "SAXParseException" in Solr's application server logs):

If you check this class: http://grepcode.com/file/repo1.maven.org/maven2/org.apache.solr/solr-core/4.3.0/org/apache/solr/servlet/SolrDispatchFilter.java/

You will see that it will return HTTP 503 if it can't initialize any cores

if (this.cores == null) {
    ((HttpServletResponse)response).sendError( 503, "Server is shutting down" );
    return;
}

This is, IMHO, kind of misleading but, this basically means that you don't have any cores/shards in your SolrCloud (i.e., someone might be messing with your Zookeeper servers), that was the root cause in my case, again, just sharing in case someone is facing the same problem.

AVINASH KUMAR · Accepted Answer · 2020-04-01 13:04:49Z

0

No disk space left out on my slave node. I cleared space and restarted tomcat, issue resolved.

Note :- check catalina.out log file, you will get exact error.

answered Apr 1, 2020 at 13:04

AVINASH KUMAR

1

Add a comment |

Stack Exchange Network

How to fix Solr - Server is shutting down issue?

6 Answers 6

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
windows-server-2008
windows-server-2008-r2
tomcat
solr
tomcat7
.

Hot Network Questions

How to fix Solr - Server is shutting down issue?

6 Answers 6

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged windows-server-2008windows-server-2008-r2tomcatsolrtomcat7.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
windows-server-2008
windows-server-2008-r2
tomcat
solr
tomcat7
.