logstash is a tool for collecting and distributing log events.

learn more… | top users | synonyms

0
votes
0answers
9 views

Separating Elasticsearch read and write

Is it possible to build a 2 or 3 node Logstash/ES cluster that separates read/query and write/indexing, so complex queries don't impact log indexing? Reading here: ...
0
votes
0answers
14 views

Logstash not saving anything for Nginx

I have a rather simple setup for Logstash as follows: /opt/logstash/patterns/nginx: NGUSERNAME [a-zA-Z\.\@\-\+_%]+ NGUSER %{NGUSERNAME} NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} ...
0
votes
1answer
9 views

How can I plot Apache HTTPd status counts in Graphite without Statsd using Logstash?

I'd like to send Apache HTTPd log statistics, such as 200 status counts to Graphite/Carbon. Logstash looks ideal but all the examples I've seen use Statsd to act as the state counter. This means ...
1
vote
0answers
23 views

logstash alert after 1000 occurences

I am trying to make Logstash to alert me only after it receives over 1000 items within 10 minutes. I need alerts in both Hipchat and PagerDuty. My config seems reasonable, but does not work as ...
0
votes
1answer
34 views

Zabbix Log alerting through Logstash

I have a problem with Zabbix and logstash alerts. I am currently sending logstash logs that are errors or warnings to zabbix via their plugin. However on Zabbix. I want to create a trigger for the ...
0
votes
0answers
6 views

Unable to get structured data in logged messages even after adding mmpstrucdata and mmrfc5424addhmac?(RFC5424)

I am trying to log messages with structured data . But it is showing null value for structured data. I am working with rsyslog 8.9.0.Can someone tell me either i need to load some module or modify ...
1
vote
1answer
48 views

How can I parse an nginx msec field in logstash?

I'm trying to handle an nginx access log in logstash. In order to get millisecond accuracy for my timestamps, I'm using the $msec variable. This means that timestamps will be of the form ...
1
vote
1answer
56 views

Bosun - Logstash Connectivity [closed]

I am in the process of tying up Bosun with my existing ELK setup. However after referring to the documents in Bosun I am not able to identify how exactly the connectivity works. is it something like ...
0
votes
1answer
204 views

Passing JSON application log to remote LogStash via NXLog on Windows

I have been trying to pass logs from a windows application which are already formatted in JSON to logstash via NXlog. When I have NXLOG send the file to Logstash, I get blasted with errors in the ...
2
votes
1answer
56 views

getting logs from systemd unit into flat files and logstash

I'm running an application under systemd within CentOS 7. It logs to stdout and systemd is capturing that into journalctl just fine. I'd like to also: get a rotated text log file also saved to the ...
1
vote
0answers
56 views

How to figure out why some characters go missing in rsyslog -> logstash pipeline

I am trying to figure out why some messages received by logstash have some text missing from them. I am sending the logs from an Untangle NGFW (11.0 - licensed) to logstash so that I can keep ...
2
votes
0answers
60 views

Elasticsearch dies when Logstash attempts to write data

I've got a Raspberry Pi 2 (latest Raspbian as of Apr 2015) setup that last week was running both ElasticSearch and Logstash on a test network (not a straightforward setup, but it was stable for over a ...
2
votes
1answer
273 views

nginx error log Grok pattern

I am having trouble getting the following nginx error log message to parse in the grok debugger. I have a feeling there is a stupid trick that I should use but can't figure out what it may be. ...
1
vote
1answer
78 views

Logstash continued grokking after _grokparsefailure

Goodday, (using logstash 1.4.2) I'm trying to do something like: filter { if type == "feed" { grok { match => [ "message", "%{COMBINEDAPACHELOGS}" ] add_tag => [ "grokked", ...
0
votes
1answer
161 views

logstash multiline log for a mysql query

I'm looking to push logs from mysql-proxy lua's script into lostash. An example log might be [2015-03-09 11:13:47] USER:username IP:10.102.51.134:41420 DB:dbName Query: -- One Pager Trends -- ...
0
votes
1answer
191 views

ESX performance stats in Logstash

Dose anyone know if it is possible to collect ESX Performance Stats in LogStash/ELK stack? looking to see if I can collect ESXTOP data for use in the ELK stack.
1
vote
2answers
175 views

How can I parse a human-readable byte count in Logstash?

I'm dealing with log files containing parts such as: 538,486K of 1,048,576K These represent memory use (Java heap space) rendered in a human-readable format. I would like to track those numbers in ...
0
votes
1answer
190 views

Logstash Grok Pattern

First of all I apologize for this,I am pretty bad in regular-expression and try to wrote custom pattern(as I am unable to find something in existing grok pattern or may be I am missing something) for ...
0
votes
1answer
268 views

Can't get logstash to work with upstart script

I've installed logstash on my headless server but I'm having a few problems getting it to work from the upstart script with sudo service logstash start. This is the output I get from ...
0
votes
0answers
206 views

Install logstash forwarder

Any body please help me how to install logstash forwarder in 12.04 ubuntu version? I followed logstash book to install forwarder I got some ruby error these are the steps i followed $ sudo ...
0
votes
1answer
116 views

IIS 7, LogStash and flushing IIS logs

I'm using LogStash to import IIS logs into ElasticSearch. It's done by pointing the LogStash to the folder with IIS logs which contains several log files each per day. Currently I don't use the ...
6
votes
2answers
201 views

Sending Subversion logs to Logstash

My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own ...
2
votes
1answer
352 views

Elasticsearch performance tuning

In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge. Logstash conf ...
0
votes
1answer
434 views

Logstash: Failed to flush outgoing items

First of all, I'm a total newbie to logstash. Despite the fact that I've managed to achieve some basic logging (trying to parse an apache log file without the build in COMBINEDAPACHELOG). However, I ...
0
votes
0answers
56 views

How parse a multiple objects serialized in log with logstash and grok

I'm parsing some log files with grok, but I have the following trace: [es.org.some.Object[ attr1=53212 searchTags=[love, rock] ]] [es.org.some.Object[ attr1=54612 searchTags=[love, peace, rock] ...
0
votes
1answer
319 views

Logststash forwarder read error looking for ack: EOF

I am receiving this error message every time logstash-forwarder has a new log to ship. Eventually the connection gets made and logstash-forwarder output confirms that the events have been processed ...
1
vote
0answers
64 views

Syslog connection to Logstash not resetting

We have a setup with one Logstash server (1.4.2, using embedded Elasticsearch) to receive logs from a number of other clients (via TCP and from their rsyslogd instances). One issue that now happens ...
1
vote
2answers
104 views

Elasticsearch index much larger than the actual size of the logs it indexed?

I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like ...
0
votes
1answer
70 views

Is there a well-known port for logstash-forwarder?

I'm setting up a few logstash-forwarder agents which will send logs to a central Logstash server, and need to choose a port for Logstash to listen on. After doing some searching, there doesn't seem to ...
0
votes
0answers
101 views

Install logstash using the Deb file

I have installed Java 7 and Java 8 on my machine and java -version returns the correct version when I change the environment variables. When I go to install Logstash using the deb I receive the follow ...
-1
votes
1answer
176 views

Why doesn't logstash grab or index the files from the mapped drive?

I don't understand why logstash is so finicky with network resources. I shared a folder on another machine and then mapped it as Z: under Windows Explorer. I've verified the path and everything. I can ...
1
vote
1answer
61 views

How does Logstash internally work with messages/events?

I'm new to Logstash, and I'm trying to understand how inputs, filters and outputs work together. I get that there's a variety of inputs, filters and outputs, but what I don't get is how Logstash works ...
2
votes
1answer
172 views

logstash timestamp on year rollover

We use logstash to store/search logs from our mail servers. I noticed today that we didn't have any indices from this year (2015). Quick investigation showed that current logs were being stored as ...
-1
votes
1answer
68 views

Statistical information regarding scaling ElasticSearch [duplicate]

I have a set-up with several Logstash nodes sending input to ElasticSearch, and have a kibana server which allows me to visualize this. The current infrastructure is pretty simplistic and is on ...
0
votes
0answers
275 views

Kibana not finding local files e.g. require JS, CSS

I'm new to linux/ubuntu and the ELK stack and having an issue getting kibana to work. I've unpacked all three into this folder and started kibana and elastic search: cd /home/ubuntu ls ...
1
vote
0answers
1k views

logstash / kibana can't connect to instance

I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries. However.. today when I went to my logstash page, I see ...
1
vote
1answer
174 views

Logstash gradually losing sockets

Using logstash 1.4.2 pushing to an elastic search 1.1.1 instance on a separate server. Pulling log items from AWS SQS. Everything works fine the first several hours or so, but then the following ...
2
votes
1answer
1k views

How to send Windows Performance counters to Logstash + Kibana?

I would like to setup monitoring for system resources on my Windows servers. I've noticed a common configuration in Linux is to use collectd daemon to get system metrics information. From collectd ...
0
votes
1answer
992 views

kibana: no indices error

None of the existing answers helped, so here a new question. Usecase: Redirecting syslog (or) monitoring static file. I have successfully installed logstash (1.4.2), elasticsearch(1.1.1) and ...
2
votes
2answers
1k views

Logstash Forwarder doesn't start up with chkconfig in CentOS 5

I have set up logstash-forwarder on a CentOS 5 machine installing it from this RPM: http://download.elasticsearch.org/logstash-forwarder/packages/logstash-forwarder-0.3.1-1.x86_64.rpm When I went ...
0
votes
0answers
44 views

Indexing individual cookies with Logstash

I'm working on some analysis of Apache logs. Our logs happen to capture the cookies the browser communicates it has and we log it with the request. Right now I can import that long string fine, ...
0
votes
1answer
118 views

install zeromq for logstash in Solaris 5.10

used this: http://zeromq.org/distro:debian to get zeromq in Ubuntu to use it with logstash. Want to achieve same within Solaris. Read that zeromq supports Solaris but only provides a tarball. How to ...
0
votes
1answer
397 views

Statistics of logstash memory consumption [duplicate]

I want to install it on a really small server and I want to see if it will be capable of running it with some dignity. So, does anyone have any memory and CPU consumption statistics of running ...
0
votes
0answers
280 views

Logstash/collectd buffer to files on network fault

I have read this: http://logstash.net/docs/1.4.2/life-of-an-event An example setup: collectd ( on local machine ) ----[via UDP]---> logstash (on local machine) ---[via elasticsearch output ...
2
votes
1answer
3k views

Logstash-forwarder is throwing SSL errors

I got this task handed over to by my colleage and this is the background. He got ELK (Elasticsearch, Logstash and Kibana) stack working with our RHEL 6.2 servers, by using the regular method of ...
1
vote
2answers
387 views

Logstash not reading logs unless world-readable

I recently set up Logstash with Kibana and Elasticsearch, but am having a problem whereby it's not reading logs unless the directory permissions are world read/execute (and the files at least world ...
0
votes
1answer
690 views

What software can I use to retrieve log files from multiple remote servers via SSH? [closed]

I know I can do this using scripts, but I'd like to know if there is such a thing as a "ready made" software which lets me configure the host (and file paths) which I'd like to fetch (at regular ...
0
votes
0answers
84 views

Logstash from RPM fails to run under nice command

I downloaded the Logstash RPM logtsash-1.4.2-1_2c0f5a1.noarch from the official site, but have been having problems getting it to run from the init script. I get no log data passed to ...
0
votes
1answer
318 views

RSyslog sends logs to logstash with host as IP not FQDN

I am having a issue. I am trying to send apache/nginx logs to logstash server. The problem is that rsyslog sends host as IP not the servers FQDN. Solutions that I have tried: PreserveFQDN on (did ...
2
votes
1answer
6k views

Logstash, Kibana and email alerts

I am trying to setup email alerts using logstash. Right now it emails me EVERY time the pattern "Error" is parsed into my log file which can lead to a lot of unnecessary emails. I'd like to create a ...