logstash is a tool for collecting and distributing log events.

learn more… | top users | synonyms

1
vote
0answers
35 views

How to figure out why some characters go missing in rsyslog -> logstash pipeline

I am trying to figure out why some messages received by logstash have some text missing from them. I am sending the logs from an Untangle NGFW (11.0 - licensed) to logstash so that I can keep ...
2
votes
0answers
29 views

Elasticsearch dies when Logstash attempts to write data

I've got a Raspberry Pi 2 (latest Raspbian as of Apr 2015) setup that last week was running both ElasticSearch and Logstash on a test network (not a straightforward setup, but it was stable for over a ...
0
votes
0answers
87 views

nginx error log Grok pattern

I am having trouble getting the following nginx error log message to parse in the grok debugger. I have a feeling there is a stupid trick that I should use but can't figure out what it may be. ...
0
votes
0answers
30 views

Logstash continued grokking after _grokparsefailure

Goodday, (using logstash 1.4.2) I'm trying to do something like: filter { if type == "feed" { grok { match => [ "message", "%{COMBINEDAPACHELOGS}" ] add_tag => [ "grokked", ...
0
votes
1answer
76 views

logstash multiline log for a mysql query

I'm looking to push logs from mysql-proxy lua's script into lostash. An example log might be [2015-03-09 11:13:47] USER:username IP:10.102.51.134:41420 DB:dbName Query: -- One Pager Trends -- ...
0
votes
1answer
127 views

ESX performance stats in Logstash

Dose anyone know if it is possible to collect ESX Performance Stats in LogStash/ELK stack? looking to see if I can collect ESXTOP data for use in the ELK stack.
1
vote
2answers
103 views

How can I parse a human-readable byte count in Logstash?

I'm dealing with log files containing parts such as: 538,486K of 1,048,576K These represent memory use (Java heap space) rendered in a human-readable format. I would like to track those numbers in ...
0
votes
1answer
113 views

Logstash Grok Pattern

First of all I apologize for this,I am pretty bad in regular-expression and try to wrote custom pattern(as I am unable to find something in existing grok pattern or may be I am missing something) for ...
0
votes
1answer
161 views

Can't get logstash to work with upstart script

I've installed logstash on my headless server but I'm having a few problems getting it to work from the upstart script with sudo service logstash start. This is the output I get from ...
0
votes
0answers
163 views

Install logstash forwarder

Any body please help me how to install logstash forwarder in 12.04 ubuntu version? I followed logstash book to install forwarder I got some ruby error these are the steps i followed $ sudo ...
0
votes
0answers
54 views

IIS 7, LogStash and flushing IIS logs

I'm using LogStash to import IIS logs into ElasticSearch. It's done by pointing the LogStash to the folder with IIS logs which contains several log files each per day. Currently I don't use the ...
6
votes
2answers
168 views

Sending Subversion logs to Logstash

My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own ...
2
votes
1answer
192 views

Elasticsearch performance tuning

In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge. Logstash conf ...
0
votes
1answer
232 views

Logstash: Failed to flush outgoing items

First of all, I'm a total newbie to logstash. Despite the fact that I've managed to achieve some basic logging (trying to parse an apache log file without the build in COMBINEDAPACHELOG). However, I ...
0
votes
0answers
49 views

How parse a multiple objects serialized in log with logstash and grok

I'm parsing some log files with grok, but I have the following trace: [es.org.some.Object[ attr1=53212 searchTags=[love, rock] ]] [es.org.some.Object[ attr1=54612 searchTags=[love, peace, rock] ...
0
votes
1answer
226 views

Logststash forwarder read error looking for ack: EOF

I am receiving this error message every time logstash-forwarder has a new log to ship. Eventually the connection gets made and logstash-forwarder output confirms that the events have been processed ...
1
vote
0answers
51 views

Syslog connection to Logstash not resetting

We have a setup with one Logstash server (1.4.2, using embedded Elasticsearch) to receive logs from a number of other clients (via TCP and from their rsyslogd instances). One issue that now happens ...
2
votes
1answer
85 views

Elasticsearch index much larger than the actual size of the logs it indexed?

I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like ...
0
votes
1answer
55 views

Is there a well-known port for logstash-forwarder?

I'm setting up a few logstash-forwarder agents which will send logs to a central Logstash server, and need to choose a port for Logstash to listen on. After doing some searching, there doesn't seem to ...
0
votes
0answers
84 views

Install logstash using the Deb file

I have installed Java 7 and Java 8 on my machine and java -version returns the correct version when I change the environment variables. When I go to install Logstash using the deb I receive the follow ...
0
votes
0answers
110 views

Why doesn't logstash grab or index the files from the mapped drive?

I don't understand why logstash is so finicky with network resources. I shared a folder on another machine and then mapped it as Z: under Windows Explorer. I've verified the path and everything. I can ...
0
votes
0answers
41 views

How does Logstash internally work with messages/events?

I'm new to Logstash, and I'm trying to understand how inputs, filters and outputs work together. I get that there's a variety of inputs, filters and outputs, but what I don't get is how Logstash works ...
2
votes
1answer
137 views

logstash timestamp on year rollover

We use logstash to store/search logs from our mail servers. I noticed today that we didn't have any indices from this year (2015). Quick investigation showed that current logs were being stored as ...
-1
votes
1answer
59 views

Statistical information regarding scaling ElasticSearch [duplicate]

I have a set-up with several Logstash nodes sending input to ElasticSearch, and have a kibana server which allows me to visualize this. The current infrastructure is pretty simplistic and is on ...
0
votes
0answers
234 views

Kibana not finding local files e.g. require JS, CSS

I'm new to linux/ubuntu and the ELK stack and having an issue getting kibana to work. I've unpacked all three into this folder and started kibana and elastic search: cd /home/ubuntu ls ...
1
vote
0answers
1k views

logstash / kibana can't connect to instance

I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries. However.. today when I went to my logstash page, I see ...
1
vote
1answer
148 views

Logstash gradually losing sockets

Using logstash 1.4.2 pushing to an elastic search 1.1.1 instance on a separate server. Pulling log items from AWS SQS. Everything works fine the first several hours or so, but then the following ...
1
vote
1answer
806 views

How to send Windows Performance counters to Logstash + Kibana?

I would like to setup monitoring for system resources on my Windows servers. I've noticed a common configuration in Linux is to use collectd daemon to get system metrics information. From collectd ...
0
votes
1answer
746 views

kibana: no indices error

None of the existing answers helped, so here a new question. Usecase: Redirecting syslog (or) monitoring static file. I have successfully installed logstash (1.4.2), elasticsearch(1.1.1) and ...
2
votes
2answers
1k views

Logstash Forwarder doesn't start up with chkconfig in CentOS 5

I have set up logstash-forwarder on a CentOS 5 machine installing it from this RPM: http://download.elasticsearch.org/logstash-forwarder/packages/logstash-forwarder-0.3.1-1.x86_64.rpm When I went ...
0
votes
0answers
35 views

Indexing individual cookies with Logstash

I'm working on some analysis of Apache logs. Our logs happen to capture the cookies the browser communicates it has and we log it with the request. Right now I can import that long string fine, ...
0
votes
1answer
97 views

install zeromq for logstash in Solaris 5.10

used this: http://zeromq.org/distro:debian to get zeromq in Ubuntu to use it with logstash. Want to achieve same within Solaris. Read that zeromq supports Solaris but only provides a tarball. How to ...
0
votes
1answer
280 views

Statistics of logstash memory consumption [duplicate]

I want to install it on a really small server and I want to see if it will be capable of running it with some dignity. So, does anyone have any memory and CPU consumption statistics of running ...
0
votes
0answers
244 views

Logstash/collectd buffer to files on network fault

I have read this: http://logstash.net/docs/1.4.2/life-of-an-event An example setup: collectd ( on local machine ) ----[via UDP]---> logstash (on local machine) ---[via elasticsearch output ...
2
votes
1answer
2k views

Logstash-forwarder is throwing SSL errors

I got this task handed over to by my colleage and this is the background. He got ELK (Elasticsearch, Logstash and Kibana) stack working with our RHEL 6.2 servers, by using the regular method of ...
1
vote
1answer
322 views

Logstash not reading logs unless world-readable

I recently set up Logstash with Kibana and Elasticsearch, but am having a problem whereby it's not reading logs unless the directory permissions are world read/execute (and the files at least world ...
0
votes
1answer
552 views

What software can I use to retrieve log files from multiple remote servers via SSH? [closed]

I know I can do this using scripts, but I'd like to know if there is such a thing as a "ready made" software which lets me configure the host (and file paths) which I'd like to fetch (at regular ...
0
votes
0answers
76 views

Logstash from RPM fails to run under nice command

I downloaded the Logstash RPM logtsash-1.4.2-1_2c0f5a1.noarch from the official site, but have been having problems getting it to run from the init script. I get no log data passed to ...
0
votes
1answer
262 views

RSyslog sends logs to logstash with host as IP not FQDN

I am having a issue. I am trying to send apache/nginx logs to logstash server. The problem is that rsyslog sends host as IP not the servers FQDN. Solutions that I have tried: PreserveFQDN on (did ...
2
votes
1answer
4k views

Logstash, Kibana and email alerts

I am trying to setup email alerts using logstash. Right now it emails me EVERY time the pattern "Error" is parsed into my log file which can lead to a lot of unnecessary emails. I'd like to create a ...
1
vote
1answer
464 views

How to include a server identifier in nxlog output and reference it in logstash

If I have nxlog running on multiple IIS servers (say web1, web2, web3, web4). How can I add an identifier to the nxlog output which is being sent to logstash. Then within logstash I want to create a ...
0
votes
0answers
228 views

logstash output to a webservice from Symfony

Good Morning, I would like to send logstash output to a web service I created with Symfony. So far my web service work, I used php to try to send a log. Quick example in PHP $client = new ...
2
votes
1answer
3k views

Logstash parsing xml document containing multiple log entries

I'm currently evaluating whether logstash and elasticsearch are useful for our use-case. What I have is a log file containing multiple entries which is of the form <root> <entry> ...
1
vote
1answer
927 views

parse old IIS logs with logstash

I have moved to logstash (combined with elastic search and Kibana) from Awsats. However I need to import/parse a few old IIS log files using logstash. The files has been moved to logstash server. ...
0
votes
1answer
1k views

Numerical range in Kibana 3 not working

My tomcat access logs have last field as "%D - Time taken to process the request, in millis". I am trying to filter the logs which took more than some number of milliseconds, but it is showing all the ...
0
votes
1answer
737 views

Inputting bulk logs in Logstash (stored in different folders) using the “file” plugin without specifying individual filters

So I have a question about the basic functionality of logstash. I have many different .L01 files which I have exported into log files. Each .L01 file represents a host and the export function dumps it ...
3
votes
1answer
1k views

Using Logstash as shipper?

We are shipping logs from servers and using THE logstash on each server for shipping. So we read logs from the glob "/root/Desktop/Logstash-Input/**/*_log". input { file{ ...
7
votes
4answers
6k views

Failed tls handshake. Does not contain any IP SANs

I'm trying to set up logstash forwarder, but I have issues with making a proper secure channel. Trying to configure this with two ubuntu (server 14.04) machines running in virtualbox. They are 100% ...
1
vote
1answer
1k views

How to parse audit.log using logstash

I want to use logstash to collect a log file, and the format of the file was like this: type=USER_START msg=audit(1404170401.294:157): user pid=29228 uid=0 auid=0 ses=7972 ...
1
vote
1answer
419 views

Correlating data within Kibana

I'm looking at aggregating data within the whole ELK stack (Elasticsearch/Logstash/Kibana) and I'm getting lots of good data from my windows systems. It's working very well, but when i get Windows ...