2

I'm having memory problems with a Solr slave instance.

The problem is that every time the slave replicates new changes from the master, the slave starts a new searcher, and apparently the old one isn't released.

Eventually the young collection is full, and it spills into the old collection (or at least that's how it appears), and it starts to do constant full garbage collections. It doesn't appear to actually be able to free any noticeable memory at this point. The full garbage collections can cause searches that usually take a few milliseconds to execute, to take up to ½-4 seconds in spikes around every 10 seconds.

And again, at some point, even the old generation is full, and then it either stops everything, and manages to free some memory, or it just appear to lock up completely, apparently never to go back to being functional.

Some system info :

The machine is an Amazon AWS m1.medium machine:
3.7GB memory
20GB HDD space
1 CPU core
2 ECU (1 ECU is the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor).

OS : Ubuntu 12.10
Java :
java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.10.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

Java servlet : Tomcat 7
Solr : 4.0

Tomcat startup parameters :

-Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp  
-Dcatalina.home=/usr/share/tomcat7  
-Dcatalina.base=/var/lib/tomcat7  
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed  
-javaagent:/spm/spm-monitor/lib/spm-monitor-solr-1.8.0-withdeps.jar=/spm/spm-monitor/conf/spm-monitor-config-[snip]-default.xml  
-Dcom.sun.management.jmxremote  
-Xloggc:/var/log/tomcat7/GCLog.txt  
-XX:+PrintGCDetails  
-XX:NewSize=500M  
-Xmx3G  
-Xms3G  
-Dsolr.solr.home=/data/solr  
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager  
-XX:+UseConcMarkSweepGC  
-Xmx128m  
-Djava.awt.headless=true  
-Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties  

the spm-monitor thing is a monitoring system from SemaText.

Solr has 4 cores, 1 for each of 4 countries, and contain product information. 3 of the cores have a small amount of documents in them, but the 4th has around 470k documents in it. That core's index currently takes up 843MB.

Here's my solrconfig.xml : http://pastebin.com/c5vHKgMF

And my schema.xml : http://pastebin.com/tFA4bmT5

I use pastebin because otherwise this post would be too big.

Here's how an output of jstat -gc [pid] 2s looks for various events :

When solr starts, it runs fine and all queries are taking normal amount of time :

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
51200.0 51200.0 51200.0  0.0   409600.0 49393.0  2633728.0   751578.5  24896.0 24736.6      4    7.271   0      0.000    7.271
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0   968965.2  24896.0 24826.1      5    7.271   0      0.000    7.271
51200.0 51200.0  0.0   51200.0 409600.0 392671.8 2633728.0  1082530.8  27584.0 27504.1      5   10.459   0      0.000   10.459
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0  1249210.0  27776.0 27578.9      7   11.822   0      0.000   11.822
51200.0 51200.0 51200.0  0.0   409600.0 90674.5  2633728.0  1484383.0  27776.0 27676.6      8   14.380   1      0.064   14.444
51200.0 51200.0 51200.0  0.0   409600.0 333155.3 2633728.0  1484383.0  28544.0 28450.1      8   14.380   1      0.064   14.444
51200.0 51200.0 51200.0  0.0   409600.0 335203.3 2633728.0  1484383.0  28544.0 28453.7      8   14.380   1      0.064   14.444
51200.0 51200.0 51200.0  0.0   409600.0 337251.4 2633728.0  1269719.9  47424.0 28459.8      8   14.380   2      0.542   14.922
51200.0 51200.0  0.0   51200.0 409600.0  1515.0  2633728.0  1347052.7  47424.0 28557.9      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 96780.5  2633728.0  1347052.7  47424.0 28634.0      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 103202.9 2633728.0  1347052.7  47424.0 28666.7      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 103203.1 2633728.0  1347052.7  47424.0 28666.7      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 109626.5 2633728.0  1347052.7  47424.0 28668.5      9   15.025   2      0.542   15.567

This is the first replication. Full GC's start to happen often after the replication, causing ½-4 seconds pauses in searches :

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
51200.0 51200.0  0.0   51200.0 409600.0 181746.1 2633728.0  1347052.7  47424.0 28745.4      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 181746.3 2633728.0  1347052.7  47424.0 28745.4      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 186884.2 2633728.0  1347052.7  47424.0 28746.4      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 186884.3 2633728.0  1347052.7  47424.0 28746.4      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 217818.1 2633728.0  1347052.7  47424.0 28754.1      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 217818.3 2633728.0  1347052.7  47424.0 28754.1      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 222957.2 2633728.0  1347052.7  47424.0 28754.8      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 222957.3 2633728.0  1347052.7  47424.0 28754.8      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 229379.7 2633728.0  1347052.7  47424.0 28756.3      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 265260.5 2633728.0  1347052.7  47424.0 30313.9      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 298923.2 2633728.0  1347052.7  47424.0 31046.5      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 301654.4 2633728.0  1347052.7  47424.0 31048.7      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 312181.6 2633728.0  1347052.7  47424.0 31051.6      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 314912.8 2633728.0  1347052.7  47424.0 31051.6      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 324156.0 2633728.0  1347052.7  47424.0 31160.7      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 329617.6 2633728.0  1347052.7  47424.0 31299.6      9   15.025   2      0.542   15.567
51200.0 51200.0  0.0   51200.0 409600.0 361996.8 2633728.0  1347052.7  47424.0 31382.4      9   15.025   2      0.542   15.567
51200.0 51200.0 30088.6  0.0   409600.0 217328.7 2633728.0  1398099.4  47424.0 32633.0     10   15.316   2      0.542   15.858
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0  1843888.2  47424.0 32853.0     12   16.800   2      0.542   17.342
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0  2049137.6  47424.0 32864.9     13   19.310   3      0.628   19.938
51200.0 51200.0  0.0   51200.0 409600.0 334292.6 2633728.0  2202213.0  47424.0 32873.8     13   22.241   3      0.628   22.869
51200.0 51200.0 51200.0 51200.0 409600.0 409600.0 2633728.0  2391020.4  47424.0 32876.1     14   22.241   3      0.628   22.869
51200.0 51200.0 51200.0  0.0   409600.0 72530.9  2633728.0  2544393.5  47424.0 32892.1     14   26.071   3      0.628   26.699
51200.0 51200.0 51200.0  0.0   409600.0 84458.3  2633728.0  2520856.9  54832.0 32898.3     14   26.071   5      0.883   26.954
51200.0 51200.0 51200.0  0.0   409600.0 90076.2  2633728.0  2520856.9  54832.0 32899.5     14   26.071   5      1.139   27.210
51200.0 51200.0 51200.0  0.0   409600.0 93651.1  2633728.0  2426346.3  55024.0 32899.9     14   26.071   6      1.387   27.458
51200.0 51200.0 51200.0  0.0   409600.0 102844.1 2633728.0  2426346.3  55024.0 32899.9     14   26.071   7      1.643   27.714
51200.0 51200.0 51200.0  0.0   409600.0 103739.1 2633728.0  2426346.3  55024.0 32901.0     14   26.071   7      1.643   27.714
51200.0 51200.0 51200.0  0.0   409600.0 133067.1 2633728.0  2426242.2  55024.0 32904.8     14   26.071   8      1.899   27.970
51200.0 51200.0 51200.0  0.0   409600.0 133946.5 2633728.0  2426227.6  55024.0 32904.9     14   26.071   9      2.175   28.245
51200.0 51200.0 51200.0  0.0   409600.0 139729.7 2633728.0  2426227.6  55024.0 32907.3     14   26.071   9      2.175   28.245
51200.0 51200.0 51200.0  0.0   409600.0 140141.2 2633728.0  2426227.6  55024.0 32907.3     14   26.071  10      2.175   28.245
51200.0 51200.0 51200.0  0.0   409600.0 146923.3 2633728.0  2426226.6  55024.0 32909.0     14   26.071  11      2.761   28.831
51200.0 51200.0 51200.0  0.0   409600.0 148190.4 2633728.0  2426226.6  55024.0 32909.3     14   26.071  11      2.761   28.831
51200.0 51200.0 51200.0  0.0   409600.0 162871.6 2633728.0  2426176.5  55024.0 32912.4     14   26.071  12      3.062   29.132
51200.0 51200.0 51200.0  0.0   409600.0 164115.7 2633728.0  2426176.5  55024.0 32912.4     14   26.071  13      3.375   29.445
51200.0 51200.0 51200.0  0.0   409600.0 169801.3 2633728.0  2426167.1  55024.0 32912.6     14   26.071  14      3.677   29.747
51200.0 51200.0 51200.0  0.0   409600.0 170464.9 2633728.0  2426167.0  55024.0 32913.0     14   26.071  15      3.978   30.048
51200.0 51200.0 51200.0  0.0   409600.0 176939.4 2633728.0  2426167.0  55024.0 32913.3     14   26.071  16      4.292   30.363
51200.0 51200.0 51200.0  0.0   409600.0 177982.8 2633728.0  2426167.0  55024.0 32916.8     14   26.071  17      4.602   30.673
51200.0 51200.0 51200.0  0.0   409600.0 207087.6 2633728.0  2426167.0  55024.0 32919.0     14   26.071  18      4.952   31.022
51200.0 51200.0 51200.0  0.0   409600.0 207915.5 2633728.0  2426167.0  55024.0 32920.0     14   26.071  19      5.289   31.359
51200.0 51200.0 51200.0  0.0   409600.0 213503.3 2633728.0  2426167.0  55024.0 32922.7     14   26.071  20      5.641   31.711
51200.0 51200.0 51200.0  0.0   409600.0 214918.9 2633728.0  2426167.0  55024.0 32922.9     14   26.071  21      6.053   32.124
51200.0 51200.0 51200.0  0.0   409600.0 221854.4 2633728.0  2426167.0  55024.0 32922.9     14   26.071  21      6.053   32.124
51200.0 51200.0 51200.0  0.0   409600.0 222649.1 2633728.0  2426167.0  55024.0 32922.9     14   26.071  23      6.763   32.834
51200.0 51200.0 51200.0  0.0   409600.0 228783.4 2633728.0  2426167.0  55024.0 32923.9     14   26.071  23      6.763   32.834
51200.0 51200.0 51200.0  0.0   409600.0 229735.2 2633728.0  2426149.5  55024.0 32924.7     14   26.071  25      7.521   33.591
51200.0 51200.0 51200.0  0.0   409600.0 235296.9 2633728.0  2426149.5  55024.0 32925.6     14   26.071  25      7.521   33.591
51200.0 51200.0 51200.0  0.0   409600.0 236130.0 2633728.0  2426149.4  55024.0 32925.6     14   26.071  27      8.260   34.331
51200.0 51200.0 51200.0  0.0   409600.0 243076.6 2633728.0  2426149.4  55024.0 32925.6     14   26.071  27      8.260   34.331
51200.0 51200.0 51200.0  0.0   409600.0 243960.8 2633728.0  2426149.4  55024.0 32926.3     14   26.071  29      9.029   35.100
51200.0 51200.0 51200.0  0.0   409600.0 273105.0 2633728.0  2426149.4  55024.0 32930.8     14   26.071  29      9.029   35.100
51200.0 51200.0 51200.0  0.0   409600.0 274353.8 2633728.0  2425918.4  55024.0 32932.0     14   26.071  31      9.436   35.507
51200.0 51200.0 51200.0  0.0   409600.0 279947.9 2633728.0  2425918.4  55024.0 32932.1     14   26.071  31      9.842   35.913
51200.0 51200.0 51200.0  0.0   409600.0 280804.8 2633728.0  2425853.6  55024.0 32932.1     14   26.071  32     10.258   36.329
51200.0 51200.0 51200.0  0.0   409600.0 288224.7 2633728.0  2425853.6  55024.0 32932.1     14   26.071  33     10.669   36.740
51200.0 51200.0 51200.0  0.0   409600.0 288924.5 2633728.0  2425801.3  55024.0 32932.1     14   26.071  34     11.086   37.157
51200.0 51200.0 51200.0  0.0   409600.0 294579.7 2633728.0  2425801.3  55024.0 32932.1     14   26.071  35     11.496   37.567
51200.0 51200.0 51200.0  0.0   409600.0 295011.3 2633728.0  2425756.8  55024.0 32932.3     14   26.071  36     11.937   38.007
51200.0 51200.0 51200.0  0.0   409600.0 300928.6 2633728.0  2425749.2  55024.0 32932.3     14   26.071  37     12.364   38.434
51200.0 51200.0 51200.0  0.0   409600.0 302173.1 2633728.0  2425749.2  55024.0 32932.3     14   26.071  38     12.794   38.865
51200.0 51200.0 51200.0  0.0   409600.0 309040.9 2633728.0  2425748.9  55024.0 32932.7     14   26.071  39     13.221   39.291
51200.0 51200.0 51200.0  0.0   409600.0 309776.0 2633728.0  2425748.9  55024.0 32932.7     14   26.071  40     13.221   39.291
51200.0 51200.0 51200.0  0.0   409600.0 338791.4 2633728.0  2425748.9  55024.0 32937.5     14   26.071  41     14.111   40.182
51200.0 51200.0 51200.0  0.0   409600.0 339363.3 2633728.0  2425748.9  55024.0 32937.5     14   26.071  41     14.111   40.182

And this is the second replication where everything almost locks up. At this point, all queries take at least 30 seconds :

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
51200.0 51200.0  0.0    0.0   409600.0 104445.7 2633728.0  2264909.9  55024.0 32909.4     15   26.071  68     33.808   59.879
51200.0 51200.0  0.0    0.0   409600.0 131077.9 2633728.0  2264909.9  55024.0 32912.0     15   26.071  69     33.902   59.973
51200.0 51200.0  0.0    0.0   409600.0 131078.1 2633728.0  2264909.9  55024.0 32912.0     15   26.071  69     33.902   59.973
51200.0 51200.0  0.0   51200.0 409600.0   0.0    2633728.0  2313467.9  55024.0 32925.0     16   26.533  70     34.030   60.563
51200.0 51200.0 51200.0  0.0   409600.0 272264.2 2633728.0  2621778.0  55024.0 32925.4     17   28.589  71     34.113   62.702
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2621778.0  55024.0 32925.7     18   28.589  72     34.113   62.702
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2621778.0  55024.0 32925.7     18   28.589  72     34.113   62.702
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2621778.0  55024.0 32925.7     18   28.589  72     34.113   62.702
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2621778.0  55024.0 32925.7     18   28.589  72     34.113   62.702
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32929.0     18   28.589  74     56.386   84.975
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32929.0     18   28.589  74     56.386   84.975
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32929.0     18   28.589  74     56.386   84.975
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32929.0     18   28.589  74     56.386   84.975
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32929.0     18   28.589  74     56.386   84.975
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.0     18   28.589  76     80.943  109.532
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.0     18   28.589  76     80.943  109.532
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.0     18   28.589  76     80.943  109.532
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.0     18   28.589  76     80.943  109.532
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.0     18   28.589  76     80.943  109.532
51200.0 51200.0 24901.3  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.0     18   28.589  77    105.639  134.229
51200.0 51200.0 51199.5  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.2     18   28.589  78    106.270  134.859
51200.0 51200.0 51199.5  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.2     18   28.589  78    106.270  134.859
51200.0 51200.0 51199.5  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.2     18   28.589  78    106.270  134.859
51200.0 51200.0 51199.5  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.2     18   28.589  78    106.270  134.859
51200.0 51200.0 37977.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32935.7     18   28.589  79    130.681  159.270
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  80    131.342  159.932
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  80    131.342  159.932
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  80    131.342  159.932
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  80    131.342  159.932
51200.0 51200.0 44939.3  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  81    155.284  183.874
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  82    155.957  184.547
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  82    155.957  184.547
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  82    155.957  184.547
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  82    155.957  184.547
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  82    155.957  184.547
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  84    181.848  210.437
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  84    181.848  210.437
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  84    181.848  210.437
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32936.2     18   28.589  84    181.848  210.437
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32922.9     18   28.589  84    181.848  210.437
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32922.9     18   28.589  85    206.950  235.539
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32922.9     18   28.589  85    206.950  235.539
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32922.9     18   28.589  85    206.950  235.539
51200.0 51200.0 51109.1  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32919.2     18   28.589  86    224.879  253.469
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32919.2     18   28.589  87    225.545  254.135
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32919.2     18   28.589  87    225.545  254.135
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32919.2     18   28.589  87    225.545  254.135
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32919.2     18   28.589  87    225.545  254.135
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32914.3     18   28.589  87    225.545  254.135
51200.0 51200.0 51194.9  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32914.5     18   28.589  88    250.623  279.213
51200.0 51200.0 51194.9  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32914.5     18   28.589  88    250.623  279.213
51200.0 51200.0 51194.9  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32901.1     18   28.589  88    250.623  279.213
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32901.1     18   28.589  89    266.103  294.693
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32901.1     18   28.589  89    266.103  294.693
51200.0 51200.0 51200.0  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32901.1     18   28.589  89    266.103  294.693
51200.0 51200.0 51184.2  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32901.1     18   28.589  91    284.683  313.272
51200.0 51200.0 51184.2  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32901.1     18   28.589  91    284.683  313.272
51200.0 51200.0 51184.2  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32901.1     18   28.589  91    284.683  313.272
51200.0 51200.0 51184.2  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32901.1     18   28.589  91    284.683  313.272
51200.0 51200.0 51184.2  0.0   409600.0 409600.0 2633728.0  2633727.9  55024.0 32901.1     18   28.589  91    284.683  313.272
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32901.3     18   28.589  92    309.958  338.547
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32901.3     18   28.589  92    309.958  338.547
51200.0 51200.0 51199.9  0.0   409600.0 409600.0 2633728.0  2633728.0  55024.0 32901.3     18   28.589  92    309.958  338.547

An output of jstat -gccause [pid] shows the following :

  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT    LGCC                 GCC
100.00   0.00 100.00 100.00  59.79     18   28.589    94  327.704  356.293 Allocation Failure   Allocation Failure

I also have a dump of the heap from one of the times where it stopped being able to even respond to searches, so if anyone wants some output from jhat i can supply that.

Does anyone have any idea why solr/tomcat/java is doing this? I can replicate it consistently, so if you need more debugging info, i can supply that as well.

1 Answer 1

0

-Xmx3G
-Xms3G
-Dsolr.solr.home=/data/solr
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-XX:+UseConcMarkSweepGC
-Xmx128m

Why do you have two max heap settings? Was it a copy paste error? Or is it the way your tomcat is configured? Also please see if you can reduce the min heap to 512M or 1G. Same size of min and max heap is generally not encouraged.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .