logstash is a tool for collecting and distributing log events.

learn more… | top users | synonyms

0
votes
1answer
11 views

Logstash S3 input plugin re-scanning all bucket objects

I am using the Logstash S3 Input plugin to process S3 access logs. The access logs are all stored in a single bucket, and there are thousands of them. I have set up the plugin to only include S3 ...
0
votes
1answer
19 views

elk stack error “unable to fetch mapping do you have indices matching the pattern”

i am trying to setup ELK stack with collectd on Ubuntu 16.04 LTS (so pretty much latest version of stack available) kibana is behind nginx proxy (followed this guide https://www.digitalocean.com/...
0
votes
1answer
16 views

Logstash Grok Filter

I'm sure there is a simple solution to this but I'm new to working Logstash. The filter I'm trying to apply is for some firewall logs and I have a field that will either have a single or two values ...
0
votes
1answer
16 views

Logstash issue matching a substring of custom field

I have the following Logstash setup. Logs are piped to Logstash from Logspout via the logspout-logstash adapter, which adds a few fields to log messages, namely the docker.image field. I am able to ...
0
votes
1answer
21 views

Can I setup logstash with windows generated certs instead of openssl?

Going through this tutorial. They use openssl to generate certs to use with logstash ie this command: sudo openssl req -subj '/CN=ELK_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -...
0
votes
0answers
72 views

“Index Patterns: Please specify a default index pattern” in Kibana

I'm trying to create a simple hello world for ELK and be able to see kibana reports via the internet. I've installed kibana, logstash, nginx and elastic search. Here's my /etc/logstash/conf.d/10-...
0
votes
1answer
42 views

How do I force a date/time format for traffic captured with tshark using “-T fields”?

I'm capturing traffic with tshark into a CSV file using "-T fields", but whatever I try I can't get a nice ISO date or even changing the timezone to UTC. The command I'm using is tshark.exe -i 2 -c ...
1
vote
0answers
42 views

object mapping for [doc.awsRegion] tried to parse field [awsRegion] as object, but found a concrete value

I am using Logstash Dynamodb PLugin to index data to be able to query it. Everything was running pretty smoothly, until one of my teammates started messing with the schema, and broke it. We can still ...
0
votes
1answer
26 views

Field type not being honor in Kibana/Elasticsearch

I'm using a logstash plug-in for a Java app using logback to forward logs on to my logstash server. I've setup a filter definition as follows: input { tcp { port => 2856 codec => ...
0
votes
2answers
173 views

Collectd --> Elasticsearch if remote-host can't connect to central elasticsearch

Goal Central storage and way to analyze performance numbers: cpu load ram usage ... Current strategy I would like to implement a setup like this: collectd logstash elasticsearch kibana Like ...
1
vote
0answers
52 views

Logstash - splitting an event into two based on objects in an included array

I'm working with logstash for the first time, and I'm trying to take JSON reports from amavisd-new in for searching and analysis. Amavisd-new is able to write the json logging to redis, and I have ...
1
vote
1answer
137 views

Sending docker logs to logstash

I have a number of CoreOS servers on Amazon AWS and would like to collect events or logs from them and forward them onto my ELK stack provider logz.io. Being a little new to the ELK stack, I'm a ...
1
vote
0answers
72 views

Redis creates many events in logstash

I'm doing some testing for a POC with logstash. When using redis to buffer messages it appears they stay in redis and continually get added to elasticsearch as new events. For example, if I restart ...
0
votes
2answers
30 views

Is is possible to pass logs to ELK stack via a ZeroMQ connection?

I have just started learning about the ELK stack. I know that Logstash can take input from several different sources via filebeat client and put that in ElasticSearch. Is it possible to pass log data ...
0
votes
0answers
12 views

How much does logstash-forwarder lag behind?

We use logstash-forwarder on around 50 systems to ship logs to our logstash collectors. Some of those systems generate a big amount of logs during peak times (firewall, auditd, ... ). I would like ...
0
votes
0answers
133 views

Logstash/Elasticsearch: frequent Connection Reset errors

I often get this in Logstash logs (every 5-10 minutes): {:timestamp=>"2016-03-08T10:53:34.880000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://xxx....
2
votes
1answer
122 views

Logstash output-http plugin error 500 when trying to send to slack webhook

I'm running a docker container with an ELK stack. Everythings working dandy and fine. I'd like to utilize the https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html plugin. I ...
-2
votes
4answers
510 views

Get logstash version

How does one get the version of Logstash? root@elk:/usr/share/elasticsearch# bin/logstash --help bash: bin/logstash: No such file or directory I have Logstash running on my system. Also. root@elk:/...
2
votes
0answers
56 views

Logstash nmap codec @timestamp match

I am using Logstash to input NMAP scan data into Elasticsearch. input{ http{ host => "127.0.0.1" port => 8000 codec => nmap } }...
0
votes
0answers
188 views

F5 High Speed Logging and Logstash GROK

Im trying to format logs straight from our F5 using HSL and Logstash. I've copied the example configuration exactly from the site as it contains the majority of the information I wanted logstash ...
0
votes
0answers
291 views

Sending PostGreSQL log data to Logstash using Packetbeat

I am trying to configure sending pgsql log data to elasticsearch. The architecture is like this: packetbeat -> logstash -> elasticsearch. After running packetbeat, logstash log says: :message "Beats ...
4
votes
1answer
86 views

Logstash with journald instead of rsyslog

I'm used to sending my logs from a server to a remote Logstash using rsyslog, with a configuration file roughly as follows (usually more specific to prevent too many logs from being sent): *.* @192....
0
votes
0answers
48 views

Using elasticsearch-logstash puppet module

I know this might be a trivial question, but I'm very new to puppet so please bear with me. I have a working puppet master/client setup on two vms. I need to use elasticsearch-logstash puppet module ...
1
vote
0answers
70 views

Visualize multiline ruby exceptions in kibana 4

I have setup the latest version of Kibana4 ElasticSearch stack. The logs are being pooled from remote app sources which are running on Ruby. I want to search for Multi Line exceptions created by ...
0
votes
1answer
88 views

Logstash conditional output to statsd - based on haproxy request URI

I have Logstash version 1.4.2 analysing haproxy logs using the HAPROXYHTTP pattern, and it is already sending counters and timing information to statsd for aggregation and onward storage in Graphite/...
1
vote
1answer
452 views

How to add AWS ELB access logs to logstash with S3 input?

I'm using an ELK stack for server monitoring. My application's access logs which are from AWS ELB are stored in AWS S3. I am trying to add them to logstash with the following input: input { s3 { ...
-1
votes
1answer
1k views

What is “io/console not supported; tty will not be manipulated”?

What is this message and how can I get rid of it? I am using ELK stack on windows 8. Every thing works just fine but this message which I get whenever I run logstash, buzzes me off. This is the ...
0
votes
0answers
112 views

Logstash Custom Grok for JSON Blocks

I've got a custom log format as follows: %DATE %TIME subscribers [{json-block}], message {json-block} heres an example: 2016-01-17 01:51:56 subscribers [{"name":"1277::1331::146::INGESTED","url":...
0
votes
1answer
757 views

Logstash grok match pattern not working even though grok constructor says it is okay

I've been building some grok patterns to parse the /var/log/secure log file and everything has been working fine. I create the grok patterns at http://grokconstructor.appspot.com/ and even then test ...
0
votes
1answer
176 views

ElasticSearch Delayed Indexing

I currently have the following setup: syslog-ng servers --> Logstash --> ElasticSearch The syslog-ng servers are load balanced and write to a SAN location where Logstash just tails the files and ...
2
votes
1answer
713 views

Parsing log4j log files with logstash

I am parsing multiline log4j logs with logstash and I've got the most curious problem. Here's an example of some related log entries: 2016-01-05 13:55:26,275 DEBUG [class] [thread] - bla 2016-01-...
0
votes
2answers
195 views

Logstash filtering syslog by host group?

I've got an Elasticsearch/Logstash/Kibana instance running, which I'm merrily stuffing with syslogs from a variety of hosts. Having built it to scale - with multiple logstash syslogd listeners, and ...
0
votes
1answer
993 views

ELK: LogStash to read log files from remote Samba-mapped network drives

I'm new to ELK, and I would like to set up a solution to index Microsoft IIS and applicative .NET logs with ES. I'm aware about different approaches: 1) [app servers: log files ➔ Logstash] ➔ [...
0
votes
1answer
71 views

Elasticsearch Cluster Configuration

Brand new to Elasticsearch. I've been doing tons of reading, but I am hoping that the experts on SO might be able to weigh in on my cluster configuration to see if there is something that I am ...
0
votes
1answer
320 views

Logstash integration with AWS Elasticsearch Service

I am using AWS Elasticsearch service to configure Elasticsearch Cluster and there is a separate server where I have installed Logstash 2.1.0 Here is my Logstash sample configuration file :- input { ...
1
vote
1answer
133 views

rsyslog cannot send logs to logstash

Here is my rsyslog.conf (nothing in /etc/rsyslog.d/ folder): #### MODULES #### $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # provides ...
8
votes
1answer
532 views

ELK Stack (Logstash, Elasticsearch and Kibana) with concurrent remote syslog server?

I'm building a log analyser service to start monitoring mainly our pfSense Firewalls, XenServer Hypervisors, FreeBSD/Linux servers and Windows servers. There's a lot of documentation on the internet ...
3
votes
1answer
153 views

Logstash - Graph IPv4 vs IPv6

I'm looking at starting to graph inbound connections to our various services (web, mail, dns etc) by IP version, ie IPv4 connections versus IPv6 connections. Currently the IPs are being stored in a ...
0
votes
1answer
402 views

Why can't the logstash syslog_pri filter see the priority in syslog messages

I'm running LS 2.0.0, and I've noticed an issue where the syslog_pri filter doesn't detect the priority at the start of my syslogs. My filter config is as follows: filter { if [type] == "relp" { ...
4
votes
1answer
96 views

logstash-forward equivalent for fluentd?

Is there something equivalent to logstash-forwarder that can ship logfiles to fluentd? I am trying to send log files from an application to a remote fluentd but have not seen whether this is ...
2
votes
1answer
215 views

Logstash event @timestamp adjustment

I have standard Windows IIS log files with event date/time stamp information and timetaken (in milliseconds). I would like to be able to adjust the event time (@timestamp) by subtracting the "...
0
votes
0answers
70 views

Logstash-forwarder Panic Centos 7

I'm trying to setup logstash forwarder, and I'm receiving the following panic when I start (using sudo service logstash-forwarder start) 2015/11/06 16:57:30.489366 --- options ------- 2015/11/06 ...
1
vote
1answer
115 views

How to format log data before forwarding them as JSON to elasticsearch?

I log all events on a system to a JSON file via syslog-ng: destination d_json { file("/var/log/all_syslog_in_json.log" perm(0666) template("{\"@timestamp\": \"$ISODATE\", \"facility\": \"$FACILITY\", ...
0
votes
1answer
175 views

Kibana: Timestamp field looks different from the Table and JSON fields in the UI

From the combined screenshot can you tell why there is a difference in timestamp for a log entry in different views in Kibana UI. Update: My timezone is +5:30 GMT, but I need the logs to show up ...
0
votes
0answers
110 views

Logstash process, created after invoking logstash.bat with task scheduler, keeps running even after task is terminated

I have dedicated machine with windows server 2012 R2 and i have to keep a logstash instance running to collect logs. For this, i have created a task scheduler task configured to invoke the provided ...
0
votes
2answers
2k views

Logstash: UNIX Epoch time not getting converted to readable format

I have setup an ELK stack and I am trying to parse squid log entries. And I am having a problem trying to convert the following UNIX/Epoc time as 1442469455.757 to a human readable format. While ...
2
votes
1answer
153 views

Running logstash under SELinux

I want to run logstash as root to allow it to read all logs (granting it access to every log is very tiresome). However, I don't want it running amok on my server, I thought about constraining it ...
1
vote
0answers
33 views

Experiences with logfile monitoring

In an IT-integrator, I'm currently trying to evangelize the use of log file monitoring of hardware installations. The idea is to use a Elasticsearch/Logstash/Kibana stack (most flexibility, open ...
0
votes
0answers
65 views

Not getting windows server hostname in logstash

Team, I have successfully installed and configured ELK to receive eventlog from windows machines, I am however not getting the name of the windows machines which are sending their logs. I am unable ...
1
vote
0answers
261 views

How do I update Logstash data to Redshift real-time?

I am looking for a way to build a pipeline for storing the logs data (I use the ELK stack) to AWS Redshift, for analytics. I have gone through the Output plugins for ElasticSearch, and found one for ...