Questions tagged [logstash]

logstash is a tool for collecting and distributing log events.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Ship Logs from application server to ELK server with beats

I am building log analyzer for production environment. My scenario is, i need to setupelasticsearch,Logstash and kibana on one centos7 server which is going to use ELK server another one is apache ...
0
votes
0answers
9 views

Adding context paths to ELK log analytics?

I'm looking to install the ELK stack on my application server by following the instructions here: I can see you can then access the various ELK services on localhost via certain ports. Is there a way ...
0
votes
1answer
16 views

Grok filter is not working properly

I have Filebeat-7.1 installed in a Debian server, this Filebeat send data from files in this Debian server to server with Logstash 7.6 , here are the files config Filebeat.yml: #=====================...
0
votes
2answers
32 views

Kibana and Logstash: When is a field not a field?

I've enabled the grok filter in our logstash configuration in order to parse logs from HAProxy using the %{HAPROXYHTTP} and %{HAPROXYTCP} patterns. This seems to work great and viewing the details for ...
0
votes
1answer
24 views

can logstash package individual log types into packages an scp them over to a central node?

I am curious whether logstash only processes logs and streams them at the event/line level or can it package all apache access logs for example and send them over zipped up to a central node?
2
votes
1answer
27 views

Repair logstash checkpoint file

We had an OutOfMemoryException in our logstash 5.6.6 which lead to the checkpoint. head file not to be written properly: ls -l /var/lib/logstash/queue/main/ total 266424 -rw-r--r-- 1 logstash ...
0
votes
1answer
70 views

What is a good approach for testing Logstash pipelines?

We have several Logstash(es) pulling data from a queue and then inserting records to Elasticsearch. Is there any good way of testing pipelines programmatically so we can include it as "unit tests" ...
0
votes
1answer
137 views

AWS ElasticSearch service does not provide Kibana users management

First I'm going to explain my current architecture. I have an EC2 instance running with a Logstash daemon, as input an http configuration and as output my AWS ElasticSearch instance. As I said before,...
0
votes
0answers
161 views

ELK - Logstash not picking up syslog events

I'm setting up a ELK cluster using Centos 8 and version 7.4 of Elasticsearch, Logstash and Kibana. My issue is with Logstash not picking up the events coming through syslog. Configuring Logstash to ...
1
vote
0answers
64 views

init: logstash main process (19281) terminated with status 1

I run Amazon Linux instance with logstash 6.4.0. [root@****** dragan]# cat /etc/issue Amazon Linux AMI release 2017.03 I wanted to upgrade logstash so I uninstalled 6.4.0 and tried to install 7.3.2. ...
0
votes
1answer
46 views

Can logstash be used as a kafka forwarder?

I am looking at the plugins for Logstash and I noticed it had both an input and output plugin for kafka. For a single topic, can Logstash be used to forward kafka topic messages to another kafka ...
0
votes
0answers
17 views

logstash not proccessing filebeat data

I have configured elk stack on a aws server and also configured filebeat on two different aws server with same config. Telnet from both filebeat server to elk server on port 5044 working fine but ...
0
votes
0answers
21 views

Trying to find source machine sending logs to ELK

So I have a machine receiving logs from multiple servers. There's a particular log that gets repeated a lot and I can't find out the source machine that is generating it. I'm fairly new to ELK, and, ...
2
votes
0answers
166 views

Logstash to aggregate logs into ElasticSearch

I am trying to aggregate linux logs using rsyslog into Logstash/ElasticSearch running in EKS. filebeat is already running in EKS to aggregate Kubernetes container logs. I have configured rsyslog ...
0
votes
0answers
32 views

Logstash not sending out any email via sendmail according to postfix logs

I have the following output config: if 'chrisdebug' in [tags] { stdout { } elasticsearch { index => "chris-debug" } email { domain => 'localhost' ...
1
vote
1answer
29 views

Documents from Elasticsearch Cluster appearing twice in Grafana

I set up an Elasticsearch cluster with one dedicated master node, two master-eligible data nodes and one coordinating node. The number of replicas is set to one. There are two pipelines in Logstash, ...
0
votes
1answer
99 views

logstash and rsyslog sends wrong hostname

I'm setting up a centralised logging system, comprising two VM's. Logger, which acts as the central syslog collector (using rsyslog) and Logstash processor; and Searcher, which acts as the ...
0
votes
0answers
162 views

Creating email alerts for kibana filters via sentil

I have a elk stack that works properly and can be monitored via the kibana dashboard. But as an improvement to the whole setup I have being researching for a reporting tool to setup with kibana. My ...
0
votes
1answer
208 views

logstash not able to upload data to elasticsearch even the pipeline started

I am using elasticsearch 7.1.1 and logstash 7.1.1. I am trying to upload a log file to elastic search using grok filter. Pipeline is getting started, but data is not getting uploaded. Here is my ...
0
votes
1answer
166 views

Monitor Java Heap on a Centos Server via Telegraf/Grafana

How to Monitor Java Heap usage on centos server using Telegraf and Grafana? This monitoring is particularly for Logstash process running on the server. I tried to use JSTAT to get data, but it has ...
0
votes
0answers
369 views

Logstash S3 Plugin with ~40 buckets very slow startup

I'm using Logstash 7.1.1 and logstash-output-s3 4.1.9. I have 38 different output locations (S3 buckets) depending on the logic. Logstash is taking nearly 30 minutes to start. [2019-06-07T01:21:21,...
0
votes
1answer
43 views

Using Elasticsearch or Logstash output?

I have been going through a few tutorials on using beats to send data to elasticsearch. I noticed that some tutorials prefer to use logstash as the output which then outputs to elasticsearch. Some ...
0
votes
1answer
102 views

Parsing or Reformatting Logs before feeding them to Splunk or Elastic Search

I have very complex log messages, that I want to reduce to the most important fields in order to save quota. The log messages are multiline and there is a lot of redundant information in them. A ...
1
vote
2answers
591 views

Line breaks in PHP's stack trace in nginx error logs disturbing logstash analysis

I am using nginx with PHP-FPM and ELK as log file analysis. When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file. ...
2
votes
0answers
2k views

Parsing JSON event in Logstash

I have log in following format, it is a plain json with nested fields. { "level": "info", "message": { "req": { "headers": { "host": "localhost:8080", ...
0
votes
1answer
75 views

logstash charset error (gelf)

I had some trouble with Logstash and Nginx (all running on Docker). On my docker-compose file i was trying to enable logging from Nginx container to Logstash: nginx: container_name: nginx image: ...
1
vote
1answer
1k views

Logstash's ElasticSearch Input Plugin should be used to output to ElasticSearch

We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch. This feature has been successfully collected by MongoDB Input Plugin in real time. However, the ElasticSearch ...
0
votes
1answer
609 views

Logstash Input MongoDB Error “No suitable driver found for jdbc:mongo://localhost:27017/test”

logasth conf "input part" input { jdbc { jdbc_driver_library => "/home/jskang/mongo-java-driver-3.2.2.jar" jdbc_driver_class => "com.mongodb.MongoClient" jdbc_connection_string =&...
0
votes
1answer
551 views

Complete information into Postfix logs

I'm succesfully using Postfix on Debian to relay my internal e-mail and I am sending logs from the mail.log file to a Logstash instance, and I am showing them on a Kibana dashboards to look for not ...
1
vote
1answer
47 views

ELK Logs from application disappearing when grok rules are enabled

We have two instances of an application (same application, different database), lets call them app1 and app2. The route the logs take is: appX runs filebeat, which picks up the applications log file ...
0
votes
1answer
153 views

Is there any version of Logstash that can send logs to both Elasticsearch 1.5 and 6.3?

I have an old Elasticsearch v1.5 domain on Amazon AWS, keeping 30 days of logs. It seems that it's not possible to upgrade it, so i have created a new domain with version ES 6.3. Now i have Logstash ...
2
votes
0answers
530 views

Logstash losing connection to Elasticsearch nodes

We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We ...
0
votes
1answer
243 views

logstash won't run as a service

Ive installed logstash v6.3.2 via yum on CentOS7. When I start it via systemctl start logstash it appears to start properly but then exits almost immediately. There are no (apparent) errors shown. (...
0
votes
1answer
1k views

Need help increasing performance of logstash

We are in the process of deploying an ELK stack and need advice and general recommendations regarding the performance of the cluster and more specifically, logstash. So the current setup we have now ...
2
votes
2answers
958 views

Windows Event Forwarding large scale with multiple domains

Let me start off explaining what I'm trying to do: We have an RMM tool installed on lots of Windows servers. It is able to send Windows Event Logs to a central store, but not in an efficient or ...
1
vote
1answer
7k views

Filebeat can't connect to logstash on another server

Filebeat (11.11.11.11) can't connect to logstash (22.22.22.22) on another server (connection reset by peer). But filebeat services from other servers can do it. Also I can connect from this server(11....
0
votes
1answer
951 views

Retrieve pfSense/freeBSD logs with elk

I am attempting to centralize logs from different systems. I installed the Elastick Stack (Elasticsearch, Logstash, Kibana) and WAZUH OSSEC on one server (named elk). I have installed the OSSEC ...
0
votes
2answers
476 views

Logstash isn't send info to elasticsearch

I have a Kibana environemtent installed, but isn't working. When I try to configure kibana, kibana says: Couldn't find any Elasticsearch data When I list indexes of elasticseach http://localhost:...
1
vote
1answer
73 views

Local Logstash failing to forward /var/log/syslog to central Logstash

I want my local instance of Logstash to forward syslog and all /var/log/* files to a remote, central instance of Logstash. The pipeline in /etc/logstash/conf/logstash-local.conf looks as follows: ...
3
votes
1answer
1k views

Why is this exclude_lines in filebeat excluding all logs?

I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-...
0
votes
2answers
246 views

Transform Apache Cookie logs to json with rsyslogd

I am logging cookie headers, some cookies may have due date, some others not, the same is tru for other cookie parameters, I need to implement a centrilaized logging, so before sending those to ...
2
votes
1answer
169 views

ELK logstash and core grok patterns

I'm evaluating the ELK stack with filebeat & logstash across a diverse range of applications/ servers. I understand the power of customising my own grok patterns for each application/log, but to ...
1
vote
1answer
183 views

logstash: Running update via RPM stepped on my logstash.yml

Using RPM to update from logstash 5.4.1 to 5.6.4 Everything went great, but the command: yum localinstall logstash-5.6.4.rpm resulted in my /etc/logstash/logstash.yml getting overwritten by the ...
0
votes
1answer
754 views

Info sent from Logstash via elastic output not showing in Kibana, but file output works fine - what am I doing wrong?

I have an "ELK stack" configuration and, at first, was doing the standard 'filebeat' syslog feeding from logstash with the elasticsearch output plugin. It worked just fine. Now I have added a TCP ...
0
votes
1answer
5k views

Auto delete elasticsearch data older than 30 days

I have setup a ELK stack to collect logs at central server. It is working perfectly. But by default it is holding elasticsearch index/data permanently. We just want to maintain the data for 30Days. ...
-1
votes
1answer
406 views

How to display filebeat logs as a single file in kibana?

I have used ELK for displaying my middleware logs in kibana. for that I am using multiline patterns which mentioned below, after those logs displaying seperatly like separate expandable. How to ...
2
votes
0answers
174 views

IIS not UTF-8 encoding logs correctly. Bug? [closed]

My server's log file is set to encode in UTF-8, it does not seem to be encoding the text correctly. For example the cookie section sometimes has a £ sign in. But in the log it is written as 0xA3 not ...
2
votes
0answers
2k views

Logstash cannot find logstash.yml although it exists

Testing logstash as instructed here. This is the command I execute (it doesn't work without the --path.settings too): /usr/share/logstash/bin/logstash -f logstash-simple.conf --path.settings /etc/...
1
vote
1answer
182 views

Logstash input log stream via port

Is there any way to read logstash raw input data that is forwarded via certain port? input settings are as below input { tcp { port => 1234 type => "test" } } and ...
1
vote
1answer
2k views

Logtash grok / multiline confusion

My real patterns are more complex but I have tried to boil the problem down to the core issue. Something I don't understand. Please try this out on http://grokconstructor.appspot.com/do/match I'm ...

1
2 3 4 5 6