Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [logstash]

logstash is a tool for collecting and distributing log events.

0
votes
0answers
10 views

Unable to start logstash service due to ruby dependency [on hold]

I am running Logstash on Raspberry Pi jessie and it is stuck in a restart loop. Syslog reports the following: Apr 20 17:00:11 raspberrypi logstash[12578]: org/jruby/RubyKernel.java:984: warning: It ...
0
votes
2answers
19 views

Line breaks in PHP's stack trace in nginx error logs disturbing logstash analysis

I am using nginx with PHP-FPM and ELK as log file analysis. When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file. ...
1
vote
0answers
38 views

Parsing JSON event in Logstash

I have log in following format, it is a plain json with nested fields. { "level": "info", "message": { "req": { "headers": { "host": "localhost:8080", ...
0
votes
0answers
35 views

Getting Apache2 combined logs into the Elastic Stack and using dashboards

I am looking into the ELK stack, and I mostly got it running. My goal would be to get the Apache2 dashboards running in Kibana, but I am unable to. This is how far I got: **On the Apache side:* I am ...
0
votes
1answer
24 views

logstash charset error (gelf)

I had some trouble with Logstash and Nginx (all running on Docker). On my docker-compose file i was trying to enable logging from Nginx container to Logstash: nginx: container_name: nginx image: ...
0
votes
0answers
81 views

Logstash: Failed to decode CEF payload

I am parsing via the cef input plugin in Logstash log files in cef format to elasticsearch. At some log files I recevice the message Failed to decode CEF payload. It seems that the problem could be ...
0
votes
0answers
35 views

Filebeat is not sending data to Logstash

I’m trying to send alerts from Snort IDS to Elasticsearch, therefore I'm using 3 technologies: Elasticsearch- https://pastebin.com/uCNMaZFJ Logstash-https://pastebin.com/zgnbbw9K Filebeat- https://...
0
votes
0answers
92 views

Logstash isn't accepting the self signed certificates and throws a sslv3 alert handshake failure

I have set up an elsatic stack in my laptop and I have configured full stack (filebeat, logstash, elasticsearch and kibana) in it. But now I need to connect filebeat and logstash in a secure manner ...
1
vote
1answer
67 views

Logstash's ElasticSearch Input Plugin should be used to output to ElasticSearch

We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch. This feature has been successfully collected by MongoDB Input Plugin in real time. However, the ElasticSearch ...
0
votes
1answer
171 views

Logstash Input MongoDB Error “No suitable driver found for jdbc:mongo://localhost:27017/test”

logasth conf "input part" input { jdbc { jdbc_driver_library => "/home/jskang/mongo-java-driver-3.2.2.jar" jdbc_driver_class => "com.mongodb.MongoClient" jdbc_connection_string =&...
0
votes
1answer
222 views

Complete information into Postfix logs

I'm succesfully using Postfix on Debian to relay my internal e-mail and I am sending logs from the mail.log file to a Logstash instance, and I am showing them on a Kibana dashboards to look for not ...
1
vote
1answer
31 views

ELK Logs from application disappearing when grok rules are enabled

We have two instances of an application (same application, different database), lets call them app1 and app2. The route the logs take is: appX runs filebeat, which picks up the applications log file ...
0
votes
1answer
81 views

Is there any version of Logstash that can send logs to both Elasticsearch 1.5 and 6.3?

I have an old Elasticsearch v1.5 domain on Amazon AWS, keeping 30 days of logs. It seems that it's not possible to upgrade it, so i have created a new domain with version ES 6.3. Now i have Logstash ...
1
vote
0answers
214 views

Logstash losing connection to Elasticsearch nodes

We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We ...
0
votes
1answer
113 views

logstash won't run as a service

Ive installed logstash v6.3.2 via yum on CentOS7. When I start it via systemctl start logstash it appears to start properly but then exits almost immediately. There are no (apparent) errors shown. (...
0
votes
1answer
308 views

Need help increasing performance of logstash

We are in the process of deploying an ELK stack and need advice and general recommendations regarding the performance of the cluster and more specifically, logstash. So the current setup we have now ...
2
votes
2answers
533 views

Windows Event Forwarding large scale with multiple domains

Let me start off explaining what I'm trying to do: We have an RMM tool installed on lots of Windows servers. It is able to send Windows Event Logs to a central store, but not in an efficient or ...
0
votes
0answers
128 views

Google Cloud VM reverse proxy for elasticstack

I have a Google Cloud VM with ELK (elasticstack) from Bitnami. Kibana dashboard is accessible with the url Also I have logstash with http plugin on which I want to execute my query from my external ...
1
vote
1answer
3k views

Filebeat can't connect to logstash on another server

Filebeat (11.11.11.11) can't connect to logstash (22.22.22.22) on another server (connection reset by peer). But filebeat services from other servers can do it. Also I can connect from this server(11....
0
votes
1answer
525 views

Retrieve pfSense/freeBSD logs with elk

I am attempting to centralize logs from different systems. I installed the Elastick Stack (Elasticsearch, Logstash, Kibana) and WAZUH OSSEC on one server (named elk). I have installed the OSSEC ...
0
votes
2answers
337 views

Logstash isn't send info to elasticsearch

I have a Kibana environemtent installed, but isn't working. When I try to configure kibana, kibana says: Couldn't find any Elasticsearch data When I list indexes of elasticseach http://localhost:...
1
vote
1answer
44 views

Local Logstash failing to forward /var/log/syslog to central Logstash

I want my local instance of Logstash to forward syslog and all /var/log/* files to a remote, central instance of Logstash. The pipeline in /etc/logstash/conf/logstash-local.conf looks as follows: ...
3
votes
1answer
734 views

Why is this exclude_lines in filebeat excluding all logs?

I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-...
0
votes
2answers
126 views

Transform Apache Cookie logs to json with rsyslogd

I am logging cookie headers, some cookies may have due date, some others not, the same is tru for other cookie parameters, I need to implement a centrilaized logging, so before sending those to ...
2
votes
1answer
136 views

ELK logstash and core grok patterns

I'm evaluating the ELK stack with filebeat & logstash across a diverse range of applications/ servers. I understand the power of customising my own grok patterns for each application/log, but to ...
1
vote
1answer
168 views

logstash: Running update via RPM stepped on my logstash.yml

Using RPM to update from logstash 5.4.1 to 5.6.4 Everything went great, but the command: yum localinstall logstash-5.6.4.rpm resulted in my /etc/logstash/logstash.yml getting overwritten by the ...
0
votes
1answer
577 views

Info sent from Logstash via elastic output not showing in Kibana, but file output works fine - what am I doing wrong?

I have an "ELK stack" configuration and, at first, was doing the standard 'filebeat' syslog feeding from logstash with the elasticsearch output plugin. It worked just fine. Now I have added a TCP ...
0
votes
1answer
3k views

Auto delete elasticsearch data older than 30 days

I have setup a ELK stack to collect logs at central server. It is working perfectly. But by default it is holding elasticsearch index/data permanently. We just want to maintain the data for 30Days. ...
-1
votes
1answer
273 views

How to display filebeat logs as a single file in kibana?

I have used ELK for displaying my middleware logs in kibana. for that I am using multiline patterns which mentioned below, after those logs displaying seperatly like separate expandable. How to ...
2
votes
0answers
130 views

IIS not UTF-8 encoding logs correctly. Bug? [closed]

My server's log file is set to encode in UTF-8, it does not seem to be encoding the text correctly. For example the cookie section sometimes has a £ sign in. But in the log it is written as 0xA3 not ...
1
vote
0answers
1k views

Logstash cannot find logstash.yml although it exists

Testing logstash as instructed here. This is the command I execute (it doesn't work without the --path.settings too): /usr/share/logstash/bin/logstash -f logstash-simple.conf --path.settings /etc/...
1
vote
1answer
121 views

Logstash input log stream via port

Is there any way to read logstash raw input data that is forwarded via certain port? input settings are as below input { tcp { port => 1234 type => "test" } } and ...
1
vote
1answer
1k views

Logtash grok / multiline confusion

My real patterns are more complex but I have tried to boil the problem down to the core issue. Something I don't understand. Please try this out on http://grokconstructor.appspot.com/do/match I'm ...
0
votes
2answers
533 views

Rsyslog not forwarding when running as service

I have 2 local VM's running CentOS 7 to simulate our production environment. 1 proxy server and 1 centralized (ELK) server. On the ELK server is running logstash to collect al syslogs from running ...
0
votes
1answer
230 views

Logstash creating many tcp connections for single hosts

Logstash: 5.3.0 nxlog: nxlog-ce-2.9.1716 Checking nestat I have like 300 established connections even though I have only about 50 hosts. I have like 40 connections for some hosts when I the client's ...
0
votes
0answers
267 views

fluentd services die immediately after they start up

Please excuse the naivety in my question, but this is not a subject I know much about at present. My company is currently running kubernetes-managed fluentd processes to push logs to logstash. These ...
2
votes
3answers
2k views

Logstash / Elasticsearch - trasform fields to lowercase in output conf

I have a standard ELK stack currently storing numerous log outputs. I'm trying to separate my indices to be source-specific. As part of my FileBeats config, some standard fields are always generated ...
1
vote
1answer
45 views

Aggregating stats from events old and new

We'd like to feed CDN logs into Graphite and aggregate numbers found in there (rate of different HTTP status-codes, average response sizes, average cache-hit ratio, etc.) However, the logs are only ...
1
vote
1answer
190 views

Logstash / Elasticsearch - Balancing index count with performance

We have a 4 data node ElasticSearch cluster: each node has 4 cores, 16GB RAM, and 160GB storage (the cluster has separate dedicated master nodes). The cluster is responsible for storing and ...
0
votes
1answer
1k views

Logstash behind the proxy doesn't work

I'm trying to launch few Logstash instances behind the proxy. I'm using nginx, but HAProxy is an option as well. worker_processes 4; events { worker_connections 1024; } http { upstream ...
0
votes
1answer
467 views

Logstash output to Azure blobstorage

I am currently importing IIS-logs into Logstash using Filebeat, and the Logstash is configured to output the documents into Elasticsearch. Now I also want to output my IIS logs to Azure storage (blob) ...
1
vote
1answer
2k views

cannot validate certificate - doesn't contain any IP SAN

I am currently in the process of installing ELK ( ElastricSearch, LogStash & Kibana) stack. My ELK server IP address is 172.29.225.32. Elastic Search config is :: # -----------------------------...
0
votes
2answers
873 views

Troubleshooting rsyslog integration with ELK stack

I'm trying to configure rsyslog to send logs to logstash on CentOS. So I was following the tutorial. However, after setting up, nothing happens. Everything started ok, not error occurred but no logs ...
0
votes
1answer
505 views

HAProxy in front of logstash broken pipe

I'm setting up ELK stacks with loadbalancing. Haproxy works fine for ES and Kibana but I'm having issues with Logstash. Here's the haproxy configuration: frontend logstash bind 0.0.0.0:5000 ...
-1
votes
1answer
499 views

logstash multiline logging with docker gelf driver

I'm trying to force logstash to not split my mulitiline logs, i'm testing it with such simple config: input { gelf { port => 5055 } filter { multiline { pattern => "^\s" what =&...
0
votes
1answer
383 views

How filter already collected info in elasticsearch5.2 on Ubuntu 16.04

Good Evening. I have a ELK stack as follows: Clients with logbeat (windows 2003, 2008 2012, and Linux Ubuntu 16.04) logstash (FreeBSD 11.0) elasticsearch5.2/kibana/nginx 10(Ubuntu 16.04) The problem ...
0
votes
2answers
2k views

ELK vs RabbitMQ for high-volume messaging/document processing?

I've been looking at the ELK stack or RabbitMQ to replace a homegrown system that ingests a large number of files (200-300 million per hour) and operates on then sends them to various locations based ...
4
votes
2answers
3k views

How to forward application logs from Docker containers to ELK

I'm trying to centralise logging in an environment that using multiple application technologies (Java, Rails and various DBs). We want to developers to bring up stacks with Docker Compose, but we ...
1
vote
1answer
1k views

Logstash multiline codec for Java stacktraces

The logstash documentation indicates that you can collapse the multiple indented lines in a Java stacktrace log entry into a single event using the multiline codec: https://www.elastic.co/guide/en/...
1
vote
1answer
415 views

Logstash Filter Conditional Not Getting Applied

I've got nxlog on my Windows servers shipping logs to Logstash (JSON-formatted). I want to clone off the security events to a SIEM, so I added the logic to catch certain Windows Event IDs: Even ...