2

I use Logstash and Elasticsearch for our squid log store and analyze.The size of the logs increase at the speed of 40Gb per day in our freeBSD ZFS storage system. the Elasticsearch gets into failure every five days and no logs can be further written.

And I tried

index.number_of_shards: 1
index.number_of_replicas: 0

But it seems no use.

attached is the snapshot of the plugs head

elasticsearch head plug status

Can anyone explain what I'm doing wrong? And which configuration should I modify.

UPDATE

The log

[2013-12-08 19:51:16,916][WARN ][index.engine.robin ] [David Cannon] [logstash-2013.12.08][0] failed engine java.lang.OutOfMemoryError: Java heap space

[2013-12-09 17:03:07,500][DEBUG][action.admin.cluster.node.info] [David Cannon] failed to execute on node [sSoHeIz5TSG8fR3IRHj_Pg] org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodeInfo] Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodeInfo]

4
  • Can you post some error logs Dec 10, 2013 at 9:25
  • Sorry I add the log there.But there are so many logs so I paste some maybe help.
    – Kindule
    Dec 10, 2013 at 9:34
  • What are the specs of the machine running Logstash? Dec 10, 2013 at 11:04
  • The Logstash running on the same machine on the Elastic-search.
    – Kindule
    Dec 10, 2013 at 22:18

1 Answer 1

1

I've seen this before, and it's all to do with the amount of Heap Space allocated to Logstash by the JVM.

You can try increasing this by providing this flag -XX:MaxHeapSize=256m to the JVM when you start it, except you should probably try setting the MaxHeapSize to something like 512m or even bigger.

Elasticsearch comes with some pretty sane defaults, but it's possible to do even more tuning, setting the size of it's heap and so on for searching.

ES and Logstash can scale to billions of events, and many terabytes of log data, with careful configuration.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .