logstash is a tool for collecting and distributing log events.

learn more… | top users | synonyms

1
vote
0answers
8 views

Put a “config-test” step into an ansible recipe

While writing this, I solved my own problem. So I'll put the question and answer in case it helps someone else (and in case someone would like to point out a better way): I'm trying to get ansible ...
0
votes
0answers
24 views

logstash doesn't create index in elasticsearch

I followed this tutorial from Digital Ocean on how to install an ELK stack on a CentOS 7 machine. Digital Ocean ELK Setup CentOS It seemed pretty good, and got me as far as having an initial ...
0
votes
0answers
49 views

Unable to fetch mapping. Do you have indices matching the pattern?

I am new to ELK and recently installed ELK on Windows Server 2008 and on Windows 7 machine. But on both the places if I open Kibana I get the message: "Unable to fetch mapping. Do you have indices ...
0
votes
1answer
18 views

Logstash tcp input not passed to elasticsearch

After successfully setting up ELK with file inputs, logstash-forwarder and seeing logs in Kibana flow from a few servers, I have attempted to set up a TCP input: tcp { codec => "json" host ...
1
vote
0answers
18 views

Logstash sometimes duplicately populating fields when using multiline filter?

The issue I have seems to be similar to https://logstash.jira.com/browse/LOGSTASH-1892, but I'm running on Logstash-shipper 1.4.2 and this page claims the issue was fixed in 1.4.1. See a screenshot ...
0
votes
1answer
24 views

Logstash matching ceph client and server separation

So I have a fresh logstash install and I am trying to deploy logstash to get a handle on the logs. I am going through and will eventually segment the logstash filters based on subsystem and ...
0
votes
0answers
25 views

Separating Elasticsearch read and write

Is it possible to build a 2 or 3 node Logstash/ES cluster that separates read/query and write/indexing, so complex queries don't impact log indexing? Reading here: ...
0
votes
0answers
27 views

Logstash not saving anything for Nginx

I have a rather simple setup for Logstash as follows: /opt/logstash/patterns/nginx: NGUSERNAME [a-zA-Z\.\@\-\+_%]+ NGUSER %{NGUSERNAME} NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} ...
0
votes
1answer
45 views

How can I plot Apache HTTPd status counts in Graphite without Statsd using Logstash?

I'd like to send Apache HTTPd log statistics, such as 200 status counts to Graphite/Carbon. Logstash looks ideal but all the examples I've seen use Statsd to act as the state counter. This means ...
1
vote
0answers
33 views

logstash alert after 1000 occurences

I am trying to make Logstash to alert me only after it receives over 1000 items within 10 minutes. I need alerts in both Hipchat and PagerDuty. My config seems reasonable, but does not work as ...
0
votes
1answer
43 views

Zabbix Log alerting through Logstash

I have a problem with Zabbix and logstash alerts. I am currently sending logstash logs that are errors or warnings to zabbix via their plugin. However on Zabbix. I want to create a trigger for the ...
0
votes
0answers
17 views

Unable to get structured data in logged messages even after adding mmpstrucdata and mmrfc5424addhmac?(RFC5424)

I am trying to log messages with structured data . But it is showing null value for structured data. I am working with rsyslog 8.9.0.Can someone tell me either i need to load some module or modify ...
1
vote
1answer
90 views

How can I parse an nginx msec field in logstash?

I'm trying to handle an nginx access log in logstash. In order to get millisecond accuracy for my timestamps, I'm using the $msec variable. This means that timestamps will be of the form ...
1
vote
1answer
78 views

Bosun - Logstash Connectivity [closed]

I am in the process of tying up Bosun with my existing ELK setup. However after referring to the documents in Bosun I am not able to identify how exactly the connectivity works. is it something like ...
0
votes
1answer
473 views

Passing JSON application log to remote LogStash via NXLog on Windows

I have been trying to pass logs from a windows application which are already formatted in JSON to logstash via NXlog. When I have NXLOG send the file to Logstash, I get blasted with errors in the ...
2
votes
1answer
112 views

getting logs from systemd unit into flat files and logstash

I'm running an application under systemd within CentOS 7. It logs to stdout and systemd is capturing that into journalctl just fine. I'd like to also: get a rotated text log file also saved to the ...
1
vote
0answers
75 views

How to figure out why some characters go missing in rsyslog -> logstash pipeline

I am trying to figure out why some messages received by logstash have some text missing from them. I am sending the logs from an Untangle NGFW (11.0 - licensed) to logstash so that I can keep ...
2
votes
0answers
82 views

Elasticsearch dies when Logstash attempts to write data

I've got a Raspberry Pi 2 (latest Raspbian as of Apr 2015) setup that last week was running both ElasticSearch and Logstash on a test network (not a straightforward setup, but it was stable for over a ...
2
votes
1answer
441 views

nginx error log Grok pattern

I am having trouble getting the following nginx error log message to parse in the grok debugger. I have a feeling there is a stupid trick that I should use but can't figure out what it may be. ...
1
vote
1answer
114 views

Logstash continued grokking after _grokparsefailure

Goodday, (using logstash 1.4.2) I'm trying to do something like: filter { if type == "feed" { grok { match => [ "message", "%{COMBINEDAPACHELOGS}" ] add_tag => [ "grokked", ...
0
votes
1answer
232 views

logstash multiline log for a mysql query

I'm looking to push logs from mysql-proxy lua's script into lostash. An example log might be [2015-03-09 11:13:47] USER:username IP:10.102.51.134:41420 DB:dbName Query: -- One Pager Trends -- ...
0
votes
1answer
261 views

ESX performance stats in Logstash

Dose anyone know if it is possible to collect ESX Performance Stats in LogStash/ELK stack? looking to see if I can collect ESXTOP data for use in the ELK stack.
1
vote
2answers
229 views

How can I parse a human-readable byte count in Logstash?

I'm dealing with log files containing parts such as: 538,486K of 1,048,576K These represent memory use (Java heap space) rendered in a human-readable format. I would like to track those numbers in ...
0
votes
1answer
250 views

Logstash Grok Pattern

First of all I apologize for this,I am pretty bad in regular-expression and try to wrote custom pattern(as I am unable to find something in existing grok pattern or may be I am missing something) for ...
0
votes
1answer
361 views

Can't get logstash to work with upstart script

I've installed logstash on my headless server but I'm having a few problems getting it to work from the upstart script with sudo service logstash start. This is the output I get from ...
0
votes
0answers
236 views

Install logstash forwarder

Any body please help me how to install logstash forwarder in 12.04 ubuntu version? I followed logstash book to install forwarder I got some ruby error these are the steps i followed $ sudo ...
0
votes
1answer
133 views

IIS 7, LogStash and flushing IIS logs

I'm using LogStash to import IIS logs into ElasticSearch. It's done by pointing the LogStash to the folder with IIS logs which contains several log files each per day. Currently I don't use the ...
5
votes
2answers
226 views

Sending Subversion logs to Logstash

My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own ...
2
votes
1answer
467 views

Elasticsearch performance tuning

In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge. Logstash conf ...
0
votes
1answer
619 views

Logstash: Failed to flush outgoing items

First of all, I'm a total newbie to logstash. Despite the fact that I've managed to achieve some basic logging (trying to parse an apache log file without the build in COMBINEDAPACHELOG). However, I ...
0
votes
0answers
69 views

How parse a multiple objects serialized in log with logstash and grok

I'm parsing some log files with grok, but I have the following trace: [es.org.some.Object[ attr1=53212 searchTags=[love, rock] ]] [es.org.some.Object[ attr1=54612 searchTags=[love, peace, rock] ...
0
votes
1answer
394 views

Logststash forwarder read error looking for ack: EOF

I am receiving this error message every time logstash-forwarder has a new log to ship. Eventually the connection gets made and logstash-forwarder output confirms that the events have been processed ...
1
vote
0answers
71 views

Syslog connection to Logstash not resetting

We have a setup with one Logstash server (1.4.2, using embedded Elasticsearch) to receive logs from a number of other clients (via TCP and from their rsyslogd instances). One issue that now happens ...
1
vote
2answers
117 views

Elasticsearch index much larger than the actual size of the logs it indexed?

I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like ...
0
votes
1answer
81 views

Is there a well-known port for logstash-forwarder?

I'm setting up a few logstash-forwarder agents which will send logs to a central Logstash server, and need to choose a port for Logstash to listen on. After doing some searching, there doesn't seem to ...
0
votes
0answers
109 views

Install logstash using the Deb file

I have installed Java 7 and Java 8 on my machine and java -version returns the correct version when I change the environment variables. When I go to install Logstash using the deb I receive the follow ...
-1
votes
1answer
243 views

Why doesn't logstash grab or index the files from the mapped drive?

I don't understand why logstash is so finicky with network resources. I shared a folder on another machine and then mapped it as Z: under Windows Explorer. I've verified the path and everything. I can ...
1
vote
1answer
72 views

How does Logstash internally work with messages/events?

I'm new to Logstash, and I'm trying to understand how inputs, filters and outputs work together. I get that there's a variety of inputs, filters and outputs, but what I don't get is how Logstash works ...
2
votes
1answer
202 views

logstash timestamp on year rollover

We use logstash to store/search logs from our mail servers. I noticed today that we didn't have any indices from this year (2015). Quick investigation showed that current logs were being stored as ...
-1
votes
1answer
78 views

Statistical information regarding scaling ElasticSearch [duplicate]

I have a set-up with several Logstash nodes sending input to ElasticSearch, and have a kibana server which allows me to visualize this. The current infrastructure is pretty simplistic and is on ...
0
votes
0answers
291 views

Kibana not finding local files e.g. require JS, CSS

I'm new to linux/ubuntu and the ELK stack and having an issue getting kibana to work. I've unpacked all three into this folder and started kibana and elastic search: cd /home/ubuntu ls ...
1
vote
0answers
1k views

logstash / kibana can't connect to instance

I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries. However.. today when I went to my logstash page, I see ...
1
vote
1answer
194 views

Logstash gradually losing sockets

Using logstash 1.4.2 pushing to an elastic search 1.1.1 instance on a separate server. Pulling log items from AWS SQS. Everything works fine the first several hours or so, but then the following ...
3
votes
2answers
1k views

How to send Windows Performance counters to Logstash + Kibana?

I would like to setup monitoring for system resources on my Windows servers. I've noticed a common configuration in Linux is to use collectd daemon to get system metrics information. From collectd ...
0
votes
1answer
1k views

kibana: no indices error

None of the existing answers helped, so here a new question. Usecase: Redirecting syslog (or) monitoring static file. I have successfully installed logstash (1.4.2), elasticsearch(1.1.1) and ...
2
votes
2answers
2k views

Logstash Forwarder doesn't start up with chkconfig in CentOS 5

I have set up logstash-forwarder on a CentOS 5 machine installing it from this RPM: http://download.elasticsearch.org/logstash-forwarder/packages/logstash-forwarder-0.3.1-1.x86_64.rpm When I went ...
0
votes
0answers
51 views

Indexing individual cookies with Logstash

I'm working on some analysis of Apache logs. Our logs happen to capture the cookies the browser communicates it has and we log it with the request. Right now I can import that long string fine, ...
0
votes
1answer
133 views

install zeromq for logstash in Solaris 5.10

used this: http://zeromq.org/distro:debian to get zeromq in Ubuntu to use it with logstash. Want to achieve same within Solaris. Read that zeromq supports Solaris but only provides a tarball. How to ...
0
votes
1answer
485 views

Statistics of logstash memory consumption [duplicate]

I want to install it on a really small server and I want to see if it will be capable of running it with some dignity. So, does anyone have any memory and CPU consumption statistics of running ...
0
votes
0answers
300 views

Logstash/collectd buffer to files on network fault

I have read this: http://logstash.net/docs/1.4.2/life-of-an-event An example setup: collectd ( on local machine ) ----[via UDP]---> logstash (on local machine) ---[via elasticsearch output ...