Questions tagged [logstash]

logstash is a tool for collecting and distributing log events.

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

Relay all received udp syslog only from any hosts to logstash, ignoring local socket collected syslog routed to /var/log files by default config

I have rsyslog server who should relay all received from network udp syslog (hundreds of hosts) to logstash syslog plugin on localhost:5514. How could I forward received from any hosts (0.0.0.0/0) ...
0
votes
1answer
33 views

How to display the logs of a VM on ELK stack which is running on another VM?

-ELK is running on localhost & successfully getting logs. -Both VM's are on same server. -OS on both VM'S is Ubuntu 18-04 LTS.
0
votes
1answer
45 views

I used elk+redis+filebeat to build the logging platform, and now after redis is replaced by codis, logstash reports???

I used elk+redis+filebeat to build the logging platform, and now after redis is replaced by codis, logstash reports: # [2020-06-18T11:20:54,146][WARN ][logstash.inputs.redis] Redis connection problem {...
0
votes
1answer
218 views

Logstash RabbitMQ output plugin

I am a new to Logstash and I want to store nginx messages in RabbitMQ queue like this: Nginx logs -(input)-> Logstash -(output)-> RabbitMQ logstash config: filter { grok { match => { ...
0
votes
1answer
398 views

Ship Logs from application server to ELK server with beats

I am building log analyzer for production environment. My scenario is, i need to setupelasticsearch,Logstash and kibana on one centos7 server which is going to use ELK server another one is apache ...
0
votes
0answers
9 views

Adding context paths to ELK log analytics?

I'm looking to install the ELK stack on my application server by following the instructions here: I can see you can then access the various ELK services on localhost via certain ports. Is there a way ...
0
votes
1answer
155 views

Grok filter is not working properly

I have Filebeat-7.1 installed in a Debian server, this Filebeat send data from files in this Debian server to server with Logstash 7.6 , here are the files config Filebeat.yml: #=====================...
0
votes
2answers
79 views

Kibana and Logstash: When is a field not a field?

I've enabled the grok filter in our logstash configuration in order to parse logs from HAProxy using the %{HAPROXYHTTP} and %{HAPROXYTCP} patterns. This seems to work great and viewing the details for ...
0
votes
1answer
24 views

can logstash package individual log types into packages an scp them over to a central node?

I am curious whether logstash only processes logs and streams them at the event/line level or can it package all apache access logs for example and send them over zipped up to a central node?
2
votes
1answer
68 views

Repair logstash checkpoint file

We had an OutOfMemoryException in our logstash 5.6.6 which lead to the checkpoint. head file not to be written properly: ls -l /var/lib/logstash/queue/main/ total 266424 -rw-r--r-- 1 logstash ...
0
votes
1answer
328 views

What is a good approach for testing Logstash pipelines?

We have several Logstash(es) pulling data from a queue and then inserting records to Elasticsearch. Is there any good way of testing pipelines programmatically so we can include it as "unit tests" ...
0
votes
1answer
405 views

AWS ElasticSearch service does not provide Kibana users management

First I'm going to explain my current architecture. I have an EC2 instance running with a Logstash daemon, as input an http configuration and as output my AWS ElasticSearch instance. As I said before,...
0
votes
0answers
647 views

ELK - Logstash not picking up syslog events

I'm setting up a ELK cluster using Centos 8 and version 7.4 of Elasticsearch, Logstash and Kibana. My issue is with Logstash not picking up the events coming through syslog. Configuring Logstash to ...
1
vote
0answers
111 views

init: logstash main process (19281) terminated with status 1

I run Amazon Linux instance with logstash 6.4.0. [root@****** dragan]# cat /etc/issue Amazon Linux AMI release 2017.03 I wanted to upgrade logstash so I uninstalled 6.4.0 and tried to install 7.3.2. ...
0
votes
1answer
78 views

Can logstash be used as a kafka forwarder?

I am looking at the plugins for Logstash and I noticed it had both an input and output plugin for kafka. For a single topic, can Logstash be used to forward kafka topic messages to another kafka ...
2
votes
0answers
195 views

Logstash to aggregate logs into ElasticSearch

I am trying to aggregate linux logs using rsyslog into Logstash/ElasticSearch running in EKS. filebeat is already running in EKS to aggregate Kubernetes container logs. I have configured rsyslog ...
1
vote
1answer
37 views

Documents from Elasticsearch Cluster appearing twice in Grafana

I set up an Elasticsearch cluster with one dedicated master node, two master-eligible data nodes and one coordinating node. The number of replicas is set to one. There are two pipelines in Logstash, ...
0
votes
1answer
215 views

logstash and rsyslog sends wrong hostname

I'm setting up a centralised logging system, comprising two VM's. Logger, which acts as the central syslog collector (using rsyslog) and Logstash processor; and Searcher, which acts as the ...
0
votes
1answer
425 views

logstash not able to upload data to elasticsearch even the pipeline started

I am using elasticsearch 7.1.1 and logstash 7.1.1. I am trying to upload a log file to elastic search using grok filter. Pipeline is getting started, but data is not getting uploaded. Here is my ...
0
votes
1answer
228 views

Monitor Java Heap on a Centos Server via Telegraf/Grafana

How to Monitor Java Heap usage on centos server using Telegraf and Grafana? This monitoring is particularly for Logstash process running on the server. I tried to use JSTAT to get data, but it has ...
0
votes
1answer
51 views

Using Elasticsearch or Logstash output?

I have been going through a few tutorials on using beats to send data to elasticsearch. I noticed that some tutorials prefer to use logstash as the output which then outputs to elasticsearch. Some ...
0
votes
1answer
149 views

Parsing or Reformatting Logs before feeding them to Splunk or Elastic Search

I have very complex log messages, that I want to reduce to the most important fields in order to save quota. The log messages are multiline and there is a lot of redundant information in them. A ...
1
vote
2answers
1k views

Line breaks in PHP's stack trace in nginx error logs disturbing logstash analysis

I am using nginx with PHP-FPM and ELK as log file analysis. When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file. ...
2
votes
0answers
3k views

Parsing JSON event in Logstash

I have log in following format, it is a plain json with nested fields. { "level": "info", "message": { "req": { "headers": { "host": "localhost:8080", ...
0
votes
1answer
107 views

logstash charset error (gelf)

I had some trouble with Logstash and Nginx (all running on Docker). On my docker-compose file i was trying to enable logging from Nginx container to Logstash: nginx: container_name: nginx image: ...
1
vote
1answer
2k views

Logstash's ElasticSearch Input Plugin should be used to output to ElasticSearch

We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch. This feature has been successfully collected by MongoDB Input Plugin in real time. However, the ElasticSearch ...
0
votes
1answer
916 views

Logstash Input MongoDB Error “No suitable driver found for jdbc:mongo://localhost:27017/test”

logasth conf "input part" input { jdbc { jdbc_driver_library => "/home/jskang/mongo-java-driver-3.2.2.jar" jdbc_driver_class => "com.mongodb.MongoClient" jdbc_connection_string =&...
0
votes
1answer
709 views

Complete information into Postfix logs

I'm succesfully using Postfix on Debian to relay my internal e-mail and I am sending logs from the mail.log file to a Logstash instance, and I am showing them on a Kibana dashboards to look for not ...
1
vote
1answer
60 views

ELK Logs from application disappearing when grok rules are enabled

We have two instances of an application (same application, different database), lets call them app1 and app2. The route the logs take is: appX runs filebeat, which picks up the applications log file ...
0
votes
1answer
170 views

Is there any version of Logstash that can send logs to both Elasticsearch 1.5 and 6.3?

I have an old Elasticsearch v1.5 domain on Amazon AWS, keeping 30 days of logs. It seems that it's not possible to upgrade it, so i have created a new domain with version ES 6.3. Now i have Logstash ...
2
votes
0answers
865 views

Logstash losing connection to Elasticsearch nodes

We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We ...
0
votes
1answer
319 views

logstash won't run as a service

Ive installed logstash v6.3.2 via yum on CentOS7. When I start it via systemctl start logstash it appears to start properly but then exits almost immediately. There are no (apparent) errors shown. (...
0
votes
1answer
1k views

Need help increasing performance of logstash

We are in the process of deploying an ELK stack and need advice and general recommendations regarding the performance of the cluster and more specifically, logstash. So the current setup we have now ...
3
votes
2answers
1k views

Windows Event Forwarding large scale with multiple domains

Let me start off explaining what I'm trying to do: We have an RMM tool installed on lots of Windows servers. It is able to send Windows Event Logs to a central store, but not in an efficient or ...
2
votes
1answer
9k views

Filebeat can't connect to logstash on another server

Filebeat (11.11.11.11) can't connect to logstash (22.22.22.22) on another server (connection reset by peer). But filebeat services from other servers can do it. Also I can connect from this server(11....
1
vote
1answer
1k views

Retrieve pfSense/freeBSD logs with elk

I am attempting to centralize logs from different systems. I installed the Elastick Stack (Elasticsearch, Logstash, Kibana) and WAZUH OSSEC on one server (named elk). I have installed the OSSEC ...
0
votes
2answers
539 views

Logstash isn't send info to elasticsearch

I have a Kibana environemtent installed, but isn't working. When I try to configure kibana, kibana says: Couldn't find any Elasticsearch data When I list indexes of elasticseach http://localhost:...
1
vote
1answer
96 views

Local Logstash failing to forward /var/log/syslog to central Logstash

I want my local instance of Logstash to forward syslog and all /var/log/* files to a remote, central instance of Logstash. The pipeline in /etc/logstash/conf/logstash-local.conf looks as follows: ...
3
votes
1answer
2k views

Why is this exclude_lines in filebeat excluding all logs?

I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-...
0
votes
1answer
302 views

Transform Apache Cookie logs to json with rsyslogd

I am logging cookie headers, some cookies may have due date, some others not, the same is tru for other cookie parameters, I need to implement a centrilaized logging, so before sending those to ...
2
votes
1answer
183 views

ELK logstash and core grok patterns

I'm evaluating the ELK stack with filebeat & logstash across a diverse range of applications/ servers. I understand the power of customising my own grok patterns for each application/log, but to ...
1
vote
1answer
193 views

logstash: Running update via RPM stepped on my logstash.yml

Using RPM to update from logstash 5.4.1 to 5.6.4 Everything went great, but the command: yum localinstall logstash-5.6.4.rpm resulted in my /etc/logstash/logstash.yml getting overwritten by the ...
0
votes
1answer
921 views

Info sent from Logstash via elastic output not showing in Kibana, but file output works fine - what am I doing wrong?

I have an "ELK stack" configuration and, at first, was doing the standard 'filebeat' syslog feeding from logstash with the elasticsearch output plugin. It worked just fine. Now I have added a TCP ...
0
votes
1answer
11k views

Auto delete elasticsearch data older than 30 days

I have setup a ELK stack to collect logs at central server. It is working perfectly. But by default it is holding elasticsearch index/data permanently. We just want to maintain the data for 30Days. ...
-1
votes
1answer
516 views

How to display filebeat logs as a single file in kibana?

I have used ELK for displaying my middleware logs in kibana. for that I am using multiline patterns which mentioned below, after those logs displaying seperatly like separate expandable. How to ...
2
votes
0answers
209 views

IIS not UTF-8 encoding logs correctly. Bug? [closed]

My server's log file is set to encode in UTF-8, it does not seem to be encoding the text correctly. For example the cookie section sometimes has a £ sign in. But in the log it is written as 0xA3 not ...
2
votes
0answers
2k views

Logstash cannot find logstash.yml although it exists

Testing logstash as instructed here. This is the command I execute (it doesn't work without the --path.settings too): /usr/share/logstash/bin/logstash -f logstash-simple.conf --path.settings /etc/...
1
vote
1answer
191 views

Logstash input log stream via port

Is there any way to read logstash raw input data that is forwarded via certain port? input settings are as below input { tcp { port => 1234 type => "test" } } and ...
1
vote
1answer
2k views

Logtash grok / multiline confusion

My real patterns are more complex but I have tried to boil the problem down to the core issue. Something I don't understand. Please try this out on http://grokconstructor.appspot.com/do/match I'm ...
1
vote
3answers
1k views

Rsyslog not forwarding when running as service

I have 2 local VM's running CentOS 7 to simulate our production environment. 1 proxy server and 1 centralized (ELK) server. On the ELK server is running logstash to collect al syslogs from running ...

1
2 3 4 5 6