I use Logstash and Elasticsearch for our squid log store and analyze.The size of the logs increase at the speed of 40Gb per day in our freeBSD ZFS storage system. the Elasticsearch gets into failure every five days and no logs can be further written.
And I tried
index.number_of_shards: 1
index.number_of_replicas: 0
But it seems no use.
attached is the snapshot of the plugs head
Can anyone explain what I'm doing wrong? And which configuration should I modify.
UPDATE
The log
[2013-12-08 19:51:16,916][WARN ][index.engine.robin ] [David Cannon] [logstash-2013.12.08][0] failed engine java.lang.OutOfMemoryError: Java heap space
[2013-12-09 17:03:07,500][DEBUG][action.admin.cluster.node.info] [David Cannon] failed to execute on node [sSoHeIz5TSG8fR3IRHj_Pg] org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodeInfo] Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodeInfo]