I'm having memory problems with a Solr slave instance.
The problem is that every time the slave replicates new changes from the master, the slave starts a new searcher, and apparently the old one isn't released.
Eventually the young collection is full, and it spills into the old collection (or at least that's how it appears), and it starts to do constant full garbage collections. It doesn't appear to actually be able to free any noticeable memory at this point. The full garbage collections can cause searches that usually take a few milliseconds to execute, to take up to ½-4 seconds in spikes around every 10 seconds.
And again, at some point, even the old generation is full, and then it either stops everything, and manages to free some memory, or it just appear to lock up completely, apparently never to go back to being functional.
Some system info :
The machine is an Amazon AWS m1.medium machine:
3.7GB memory
20GB HDD space
1 CPU core
2 ECU (1 ECU is the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor).
OS : Ubuntu 12.10
Java :
java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.10.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
Java servlet : Tomcat 7
Solr : 4.0
Tomcat startup parameters :
-Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp
-Dcatalina.home=/usr/share/tomcat7
-Dcatalina.base=/var/lib/tomcat7
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed
-javaagent:/spm/spm-monitor/lib/spm-monitor-solr-1.8.0-withdeps.jar=/spm/spm-monitor/conf/spm-monitor-config-[snip]-default.xml
-Dcom.sun.management.jmxremote
-Xloggc:/var/log/tomcat7/GCLog.txt
-XX:+PrintGCDetails
-XX:NewSize=500M
-Xmx3G
-Xms3G
-Dsolr.solr.home=/data/solr
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-XX:+UseConcMarkSweepGC
-Xmx128m
-Djava.awt.headless=true
-Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties
the spm-monitor thing is a monitoring system from SemaText.
Solr has 4 cores, 1 for each of 4 countries, and contain product information. 3 of the cores have a small amount of documents in them, but the 4th has around 470k documents in it. That core's index currently takes up 843MB.
Here's my solrconfig.xml : http://pastebin.com/c5vHKgMF
And my schema.xml : http://pastebin.com/tFA4bmT5
I use pastebin because otherwise this post would be too big.
Here's how an output of jstat -gc [pid] 2s
looks for various events :
When solr starts, it runs fine and all queries are taking normal amount of time :
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
51200.0 51200.0 51200.0 0.0 409600.0 49393.0 2633728.0 751578.5 24896.0 24736.6 4 7.271 0 0.000 7.271
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0 968965.2 24896.0 24826.1 5 7.271 0 0.000 7.271
51200.0 51200.0 0.0 51200.0 409600.0 392671.8 2633728.0 1082530.8 27584.0 27504.1 5 10.459 0 0.000 10.459
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0 1249210.0 27776.0 27578.9 7 11.822 0 0.000 11.822
51200.0 51200.0 51200.0 0.0 409600.0 90674.5 2633728.0 1484383.0 27776.0 27676.6 8 14.380 1 0.064 14.444
51200.0 51200.0 51200.0 0.0 409600.0 333155.3 2633728.0 1484383.0 28544.0 28450.1 8 14.380 1 0.064 14.444
51200.0 51200.0 51200.0 0.0 409600.0 335203.3 2633728.0 1484383.0 28544.0 28453.7 8 14.380 1 0.064 14.444
51200.0 51200.0 51200.0 0.0 409600.0 337251.4 2633728.0 1269719.9 47424.0 28459.8 8 14.380 2 0.542 14.922
51200.0 51200.0 0.0 51200.0 409600.0 1515.0 2633728.0 1347052.7 47424.0 28557.9 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 96780.5 2633728.0 1347052.7 47424.0 28634.0 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 103202.9 2633728.0 1347052.7 47424.0 28666.7 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 103203.1 2633728.0 1347052.7 47424.0 28666.7 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 109626.5 2633728.0 1347052.7 47424.0 28668.5 9 15.025 2 0.542 15.567
This is the first replication. Full GC's start to happen often after the replication, causing ½-4 seconds pauses in searches :
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
51200.0 51200.0 0.0 51200.0 409600.0 181746.1 2633728.0 1347052.7 47424.0 28745.4 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 181746.3 2633728.0 1347052.7 47424.0 28745.4 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 186884.2 2633728.0 1347052.7 47424.0 28746.4 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 186884.3 2633728.0 1347052.7 47424.0 28746.4 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 217818.1 2633728.0 1347052.7 47424.0 28754.1 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 217818.3 2633728.0 1347052.7 47424.0 28754.1 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 222957.2 2633728.0 1347052.7 47424.0 28754.8 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 222957.3 2633728.0 1347052.7 47424.0 28754.8 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 229379.7 2633728.0 1347052.7 47424.0 28756.3 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 265260.5 2633728.0 1347052.7 47424.0 30313.9 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 298923.2 2633728.0 1347052.7 47424.0 31046.5 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 301654.4 2633728.0 1347052.7 47424.0 31048.7 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 312181.6 2633728.0 1347052.7 47424.0 31051.6 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 314912.8 2633728.0 1347052.7 47424.0 31051.6 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 324156.0 2633728.0 1347052.7 47424.0 31160.7 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 329617.6 2633728.0 1347052.7 47424.0 31299.6 9 15.025 2 0.542 15.567
51200.0 51200.0 0.0 51200.0 409600.0 361996.8 2633728.0 1347052.7 47424.0 31382.4 9 15.025 2 0.542 15.567
51200.0 51200.0 30088.6 0.0 409600.0 217328.7 2633728.0 1398099.4 47424.0 32633.0 10 15.316 2 0.542 15.858
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0 1843888.2 47424.0 32853.0 12 16.800 2 0.542 17.342
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0 2049137.6 47424.0 32864.9 13 19.310 3 0.628 19.938
51200.0 51200.0 0.0 51200.0 409600.0 334292.6 2633728.0 2202213.0 47424.0 32873.8 13 22.241 3 0.628 22.869
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0 2391020.4 47424.0 32876.1 14 22.241 3 0.628 22.869
51200.0 51200.0 51200.0 0.0 409600.0 72530.9 2633728.0 2544393.5 47424.0 32892.1 14 26.071 3 0.628 26.699
51200.0 51200.0 51200.0 0.0 409600.0 84458.3 2633728.0 2520856.9 54832.0 32898.3 14 26.071 5 0.883 26.954
51200.0 51200.0 51200.0 0.0 409600.0 90076.2 2633728.0 2520856.9 54832.0 32899.5 14 26.071 5 1.139 27.210
51200.0 51200.0 51200.0 0.0 409600.0 93651.1 2633728.0 2426346.3 55024.0 32899.9 14 26.071 6 1.387 27.458
51200.0 51200.0 51200.0 0.0 409600.0 102844.1 2633728.0 2426346.3 55024.0 32899.9 14 26.071 7 1.643 27.714
51200.0 51200.0 51200.0 0.0 409600.0 103739.1 2633728.0 2426346.3 55024.0 32901.0 14 26.071 7 1.643 27.714
51200.0 51200.0 51200.0 0.0 409600.0 133067.1 2633728.0 2426242.2 55024.0 32904.8 14 26.071 8 1.899 27.970
51200.0 51200.0 51200.0 0.0 409600.0 133946.5 2633728.0 2426227.6 55024.0 32904.9 14 26.071 9 2.175 28.245
51200.0 51200.0 51200.0 0.0 409600.0 139729.7 2633728.0 2426227.6 55024.0 32907.3 14 26.071 9 2.175 28.245
51200.0 51200.0 51200.0 0.0 409600.0 140141.2 2633728.0 2426227.6 55024.0 32907.3 14 26.071 10 2.175 28.245
51200.0 51200.0 51200.0 0.0 409600.0 146923.3 2633728.0 2426226.6 55024.0 32909.0 14 26.071 11 2.761 28.831
51200.0 51200.0 51200.0 0.0 409600.0 148190.4 2633728.0 2426226.6 55024.0 32909.3 14 26.071 11 2.761 28.831
51200.0 51200.0 51200.0 0.0 409600.0 162871.6 2633728.0 2426176.5 55024.0 32912.4 14 26.071 12 3.062 29.132
51200.0 51200.0 51200.0 0.0 409600.0 164115.7 2633728.0 2426176.5 55024.0 32912.4 14 26.071 13 3.375 29.445
51200.0 51200.0 51200.0 0.0 409600.0 169801.3 2633728.0 2426167.1 55024.0 32912.6 14 26.071 14 3.677 29.747
51200.0 51200.0 51200.0 0.0 409600.0 170464.9 2633728.0 2426167.0 55024.0 32913.0 14 26.071 15 3.978 30.048
51200.0 51200.0 51200.0 0.0 409600.0 176939.4 2633728.0 2426167.0 55024.0 32913.3 14 26.071 16 4.292 30.363
51200.0 51200.0 51200.0 0.0 409600.0 177982.8 2633728.0 2426167.0 55024.0 32916.8 14 26.071 17 4.602 30.673
51200.0 51200.0 51200.0 0.0 409600.0 207087.6 2633728.0 2426167.0 55024.0 32919.0 14 26.071 18 4.952 31.022
51200.0 51200.0 51200.0 0.0 409600.0 207915.5 2633728.0 2426167.0 55024.0 32920.0 14 26.071 19 5.289 31.359
51200.0 51200.0 51200.0 0.0 409600.0 213503.3 2633728.0 2426167.0 55024.0 32922.7 14 26.071 20 5.641 31.711
51200.0 51200.0 51200.0 0.0 409600.0 214918.9 2633728.0 2426167.0 55024.0 32922.9 14 26.071 21 6.053 32.124
51200.0 51200.0 51200.0 0.0 409600.0 221854.4 2633728.0 2426167.0 55024.0 32922.9 14 26.071 21 6.053 32.124
51200.0 51200.0 51200.0 0.0 409600.0 222649.1 2633728.0 2426167.0 55024.0 32922.9 14 26.071 23 6.763 32.834
51200.0 51200.0 51200.0 0.0 409600.0 228783.4 2633728.0 2426167.0 55024.0 32923.9 14 26.071 23 6.763 32.834
51200.0 51200.0 51200.0 0.0 409600.0 229735.2 2633728.0 2426149.5 55024.0 32924.7 14 26.071 25 7.521 33.591
51200.0 51200.0 51200.0 0.0 409600.0 235296.9 2633728.0 2426149.5 55024.0 32925.6 14 26.071 25 7.521 33.591
51200.0 51200.0 51200.0 0.0 409600.0 236130.0 2633728.0 2426149.4 55024.0 32925.6 14 26.071 27 8.260 34.331
51200.0 51200.0 51200.0 0.0 409600.0 243076.6 2633728.0 2426149.4 55024.0 32925.6 14 26.071 27 8.260 34.331
51200.0 51200.0 51200.0 0.0 409600.0 243960.8 2633728.0 2426149.4 55024.0 32926.3 14 26.071 29 9.029 35.100
51200.0 51200.0 51200.0 0.0 409600.0 273105.0 2633728.0 2426149.4 55024.0 32930.8 14 26.071 29 9.029 35.100
51200.0 51200.0 51200.0 0.0 409600.0 274353.8 2633728.0 2425918.4 55024.0 32932.0 14 26.071 31 9.436 35.507
51200.0 51200.0 51200.0 0.0 409600.0 279947.9 2633728.0 2425918.4 55024.0 32932.1 14 26.071 31 9.842 35.913
51200.0 51200.0 51200.0 0.0 409600.0 280804.8 2633728.0 2425853.6 55024.0 32932.1 14 26.071 32 10.258 36.329
51200.0 51200.0 51200.0 0.0 409600.0 288224.7 2633728.0 2425853.6 55024.0 32932.1 14 26.071 33 10.669 36.740
51200.0 51200.0 51200.0 0.0 409600.0 288924.5 2633728.0 2425801.3 55024.0 32932.1 14 26.071 34 11.086 37.157
51200.0 51200.0 51200.0 0.0 409600.0 294579.7 2633728.0 2425801.3 55024.0 32932.1 14 26.071 35 11.496 37.567
51200.0 51200.0 51200.0 0.0 409600.0 295011.3 2633728.0 2425756.8 55024.0 32932.3 14 26.071 36 11.937 38.007
51200.0 51200.0 51200.0 0.0 409600.0 300928.6 2633728.0 2425749.2 55024.0 32932.3 14 26.071 37 12.364 38.434
51200.0 51200.0 51200.0 0.0 409600.0 302173.1 2633728.0 2425749.2 55024.0 32932.3 14 26.071 38 12.794 38.865
51200.0 51200.0 51200.0 0.0 409600.0 309040.9 2633728.0 2425748.9 55024.0 32932.7 14 26.071 39 13.221 39.291
51200.0 51200.0 51200.0 0.0 409600.0 309776.0 2633728.0 2425748.9 55024.0 32932.7 14 26.071 40 13.221 39.291
51200.0 51200.0 51200.0 0.0 409600.0 338791.4 2633728.0 2425748.9 55024.0 32937.5 14 26.071 41 14.111 40.182
51200.0 51200.0 51200.0 0.0 409600.0 339363.3 2633728.0 2425748.9 55024.0 32937.5 14 26.071 41 14.111 40.182
And this is the second replication where everything almost locks up. At this point, all queries take at least 30 seconds :
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
51200.0 51200.0 0.0 0.0 409600.0 104445.7 2633728.0 2264909.9 55024.0 32909.4 15 26.071 68 33.808 59.879
51200.0 51200.0 0.0 0.0 409600.0 131077.9 2633728.0 2264909.9 55024.0 32912.0 15 26.071 69 33.902 59.973
51200.0 51200.0 0.0 0.0 409600.0 131078.1 2633728.0 2264909.9 55024.0 32912.0 15 26.071 69 33.902 59.973
51200.0 51200.0 0.0 51200.0 409600.0 0.0 2633728.0 2313467.9 55024.0 32925.0 16 26.533 70 34.030 60.563
51200.0 51200.0 51200.0 0.0 409600.0 272264.2 2633728.0 2621778.0 55024.0 32925.4 17 28.589 71 34.113 62.702
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2621778.0 55024.0 32925.7 18 28.589 72 34.113 62.702
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2621778.0 55024.0 32925.7 18 28.589 72 34.113 62.702
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2621778.0 55024.0 32925.7 18 28.589 72 34.113 62.702
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2621778.0 55024.0 32925.7 18 28.589 72 34.113 62.702
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32929.0 18 28.589 74 56.386 84.975
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32929.0 18 28.589 74 56.386 84.975
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32929.0 18 28.589 74 56.386 84.975
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32929.0 18 28.589 74 56.386 84.975
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32929.0 18 28.589 74 56.386 84.975
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.0 18 28.589 76 80.943 109.532
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.0 18 28.589 76 80.943 109.532
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.0 18 28.589 76 80.943 109.532
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.0 18 28.589 76 80.943 109.532
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.0 18 28.589 76 80.943 109.532
51200.0 51200.0 24901.3 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.0 18 28.589 77 105.639 134.229
51200.0 51200.0 51199.5 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.2 18 28.589 78 106.270 134.859
51200.0 51200.0 51199.5 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.2 18 28.589 78 106.270 134.859
51200.0 51200.0 51199.5 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.2 18 28.589 78 106.270 134.859
51200.0 51200.0 51199.5 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.2 18 28.589 78 106.270 134.859
51200.0 51200.0 37977.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32935.7 18 28.589 79 130.681 159.270
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 80 131.342 159.932
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 80 131.342 159.932
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 80 131.342 159.932
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 80 131.342 159.932
51200.0 51200.0 44939.3 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 81 155.284 183.874
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 82 155.957 184.547
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 82 155.957 184.547
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 82 155.957 184.547
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 82 155.957 184.547
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 82 155.957 184.547
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 84 181.848 210.437
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 84 181.848 210.437
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 84 181.848 210.437
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32936.2 18 28.589 84 181.848 210.437
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32922.9 18 28.589 84 181.848 210.437
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32922.9 18 28.589 85 206.950 235.539
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32922.9 18 28.589 85 206.950 235.539
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32922.9 18 28.589 85 206.950 235.539
51200.0 51200.0 51109.1 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32919.2 18 28.589 86 224.879 253.469
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32919.2 18 28.589 87 225.545 254.135
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32919.2 18 28.589 87 225.545 254.135
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32919.2 18 28.589 87 225.545 254.135
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32919.2 18 28.589 87 225.545 254.135
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32914.3 18 28.589 87 225.545 254.135
51200.0 51200.0 51194.9 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32914.5 18 28.589 88 250.623 279.213
51200.0 51200.0 51194.9 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32914.5 18 28.589 88 250.623 279.213
51200.0 51200.0 51194.9 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32901.1 18 28.589 88 250.623 279.213
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32901.1 18 28.589 89 266.103 294.693
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32901.1 18 28.589 89 266.103 294.693
51200.0 51200.0 51200.0 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32901.1 18 28.589 89 266.103 294.693
51200.0 51200.0 51184.2 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32901.1 18 28.589 91 284.683 313.272
51200.0 51200.0 51184.2 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32901.1 18 28.589 91 284.683 313.272
51200.0 51200.0 51184.2 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32901.1 18 28.589 91 284.683 313.272
51200.0 51200.0 51184.2 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32901.1 18 28.589 91 284.683 313.272
51200.0 51200.0 51184.2 0.0 409600.0 409600.0 2633728.0 2633727.9 55024.0 32901.1 18 28.589 91 284.683 313.272
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32901.3 18 28.589 92 309.958 338.547
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32901.3 18 28.589 92 309.958 338.547
51200.0 51200.0 51199.9 0.0 409600.0 409600.0 2633728.0 2633728.0 55024.0 32901.3 18 28.589 92 309.958 338.547
An output of jstat -gccause [pid]
shows the following :
S0 S1 E O P YGC YGCT FGC FGCT GCT LGCC GCC
100.00 0.00 100.00 100.00 59.79 18 28.589 94 327.704 356.293 Allocation Failure Allocation Failure
I also have a dump of the heap from one of the times where it stopped being able to even respond to searches, so if anyone wants some output from jhat
i can supply that.
Does anyone have any idea why solr/tomcat/java is doing this? I can replicate it consistently, so if you need more debugging info, i can supply that as well.