Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [logstash]

logstash is a tool for collecting and distributing log events.

Filter by
Sorted by
Tagged with
0 votes
1 answer
29 views

Can logstash "pull" data?

I have two servers. Server A is running Elasticsearch and Logstash. Server B is running filebeat and is also the server which contains all the logs I'm trying to analyse. Server A is behind a firewall,...
user avatar
  • 851
0 votes
0 answers
123 views

Rsyslog only forwarding some log files to logstash

I have a client machine which forwards its log files to a central rsyslog server. I can see the logs on the central syslog server here: root@server:/var/log# ls -al /var/log/client/ total 1952 drwxr-...
user avatar
  • 101
0 votes
1 answer
68 views

How to display the logs of a VM on ELK stack which is running on another VM?

-ELK is running on localhost & successfully getting logs. -Both VM's are on same server. -OS on both VM'S is Ubuntu 18-04 LTS.
user avatar
0 votes
1 answer
60 views

I used elk+redis+filebeat to build the logging platform, and now after redis is replaced by codis, logstash reports???

I used elk+redis+filebeat to build the logging platform, and now after redis is replaced by codis, logstash reports: # [2020-06-18T11:20:54,146][WARN ][logstash.inputs.redis] Redis connection problem {...
user avatar
0 votes
1 answer
632 views

Logstash RabbitMQ output plugin

I am a new to Logstash and I want to store nginx messages in RabbitMQ queue like this: Nginx logs -(input)-> Logstash -(output)-> RabbitMQ logstash config: filter { grok { match => { ...
user avatar
0 votes
1 answer
753 views

Ship Logs from application server to ELK server with beats

I am building log analyzer for production environment. My scenario is, i need to setupelasticsearch,Logstash and kibana on one centos7 server which is going to use ELK server another one is apache ...
user avatar
  • 3
0 votes
0 answers
13 views

Adding context paths to ELK log analytics?

I'm looking to install the ELK stack on my application server by following the instructions here: I can see you can then access the various ELK services on localhost via certain ports. Is there a way ...
user avatar
0 votes
1 answer
413 views

Grok filter is not working properly

I have Filebeat-7.1 installed in a Debian server, this Filebeat send data from files in this Debian server to server with Logstash 7.6 , here are the files config Filebeat.yml: #=====================...
user avatar
  • 1
0 votes
2 answers
170 views

Kibana and Logstash: When is a field not a field?

I've enabled the grok filter in our logstash configuration in order to parse logs from HAProxy using the %{HAPROXYHTTP} and %{HAPROXYTCP} patterns. This seems to work great and viewing the details for ...
user avatar
  • 40.3k
0 votes
1 answer
27 views

can logstash package individual log types into packages an scp them over to a central node?

I am curious whether logstash only processes logs and streams them at the event/line level or can it package all apache access logs for example and send them over zipped up to a central node?
user avatar
  • 157
2 votes
1 answer
122 views

Repair logstash checkpoint file

We had an OutOfMemoryException in our logstash 5.6.6 which lead to the checkpoint. head file not to be written properly: ls -l /var/lib/logstash/queue/main/ total 266424 -rw-r--r-- 1 logstash ...
user avatar
  • 75
1 vote
1 answer
503 views

What is a good approach for testing Logstash pipelines?

We have several Logstash(es) pulling data from a queue and then inserting records to Elasticsearch. Is there any good way of testing pipelines programmatically so we can include it as "unit tests" ...
user avatar
  • 187
0 votes
1 answer
517 views

AWS ElasticSearch service does not provide Kibana users management

First I'm going to explain my current architecture. I have an EC2 instance running with a Logstash daemon, as input an http configuration and as output my AWS ElasticSearch instance. As I said before,...
user avatar
0 votes
0 answers
1k views

ELK - Logstash not picking up syslog events

I'm setting up a ELK cluster using Centos 8 and version 7.4 of Elasticsearch, Logstash and Kibana. My issue is with Logstash not picking up the events coming through syslog. Configuring Logstash to ...
user avatar
  • 257
1 vote
0 answers
132 views

init: logstash main process (19281) terminated with status 1

I run Amazon Linux instance with logstash 6.4.0. [root@****** dragan]# cat /etc/issue Amazon Linux AMI release 2017.03 I wanted to upgrade logstash so I uninstalled 6.4.0 and tried to install 7.3.2. ...
user avatar
  • 11
0 votes
1 answer
103 views

Can logstash be used as a kafka forwarder?

I am looking at the plugins for Logstash and I noticed it had both an input and output plugin for kafka. For a single topic, can Logstash be used to forward kafka topic messages to another kafka ...
user avatar
2 votes
0 answers
245 views

Logstash to aggregate logs into ElasticSearch

I am trying to aggregate linux logs using rsyslog into Logstash/ElasticSearch running in EKS. filebeat is already running in EKS to aggregate Kubernetes container logs. I have configured rsyslog ...
user avatar
  • 109
1 vote
1 answer
46 views

Documents from Elasticsearch Cluster appearing twice in Grafana

I set up an Elasticsearch cluster with one dedicated master node, two master-eligible data nodes and one coordinating node. The number of replicas is set to one. There are two pipelines in Logstash, ...
user avatar
  • 141
0 votes
1 answer
318 views

logstash and rsyslog sends wrong hostname

I'm setting up a centralised logging system, comprising two VM's. Logger, which acts as the central syslog collector (using rsyslog) and Logstash processor; and Searcher, which acts as the ...
user avatar
0 votes
1 answer
654 views

logstash not able to upload data to elasticsearch even the pipeline started

I am using elasticsearch 7.1.1 and logstash 7.1.1. I am trying to upload a log file to elastic search using grok filter. Pipeline is getting started, but data is not getting uploaded. Here is my ...
user avatar
0 votes
1 answer
286 views

Monitor Java Heap on a Centos Server via Telegraf/Grafana

How to Monitor Java Heap usage on centos server using Telegraf and Grafana? This monitoring is particularly for Logstash process running on the server. I tried to use JSTAT to get data, but it has ...
user avatar
  • 827
0 votes
1 answer
79 views

Using Elasticsearch or Logstash output?

I have been going through a few tutorials on using beats to send data to elasticsearch. I noticed that some tutorials prefer to use logstash as the output which then outputs to elasticsearch. Some ...
user avatar
  • 403
0 votes
1 answer
174 views

Parsing or Reformatting Logs before feeding them to Splunk or Elastic Search

I have very complex log messages, that I want to reduce to the most important fields in order to save quota. The log messages are multiline and there is a lot of redundant information in them. A ...
user avatar
  • 131
1 vote
2 answers
2k views

Line breaks in PHP's stack trace in nginx error logs disturbing logstash analysis

I am using nginx with PHP-FPM and ELK as log file analysis. When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file. ...
user avatar
  • 229
2 votes
0 answers
3k views

Parsing JSON event in Logstash

I have log in following format, it is a plain json with nested fields. { "level": "info", "message": { "req": { "headers": { "host": "localhost:8080", ...
user avatar
  • 21
0 votes
1 answer
180 views

logstash charset error (gelf)

I had some trouble with Logstash and Nginx (all running on Docker). On my docker-compose file i was trying to enable logging from Nginx container to Logstash: nginx: container_name: nginx image: ...
user avatar
1 vote
1 answer
2k views

Logstash's ElasticSearch Input Plugin should be used to output to ElasticSearch

We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch. This feature has been successfully collected by MongoDB Input Plugin in real time. However, the ElasticSearch ...
user avatar
0 votes
1 answer
1k views

Logstash Input MongoDB Error “No suitable driver found for jdbc:mongo://localhost:27017/test”

logasth conf "input part" input { jdbc { jdbc_driver_library => "/home/jskang/mongo-java-driver-3.2.2.jar" jdbc_driver_class => "com.mongodb.MongoClient" jdbc_connection_string =&...
user avatar
0 votes
1 answer
947 views

Complete information into Postfix logs

I'm succesfully using Postfix on Debian to relay my internal e-mail and I am sending logs from the mail.log file to a Logstash instance, and I am showing them on a Kibana dashboards to look for not ...
user avatar
  • 1,753
1 vote
1 answer
90 views

ELK Logs from application disappearing when grok rules are enabled

We have two instances of an application (same application, different database), lets call them app1 and app2. The route the logs take is: appX runs filebeat, which picks up the applications log file ...
user avatar
  • 1,627
0 votes
1 answer
198 views

Is there any version of Logstash that can send logs to both Elasticsearch 1.5 and 6.3?

I have an old Elasticsearch v1.5 domain on Amazon AWS, keeping 30 days of logs. It seems that it's not possible to upgrade it, so i have created a new domain with version ES 6.3. Now i have Logstash ...
user avatar
2 votes
0 answers
1k views

Logstash losing connection to Elasticsearch nodes

We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We ...
user avatar
0 votes
1 answer
484 views

logstash won't run as a service

Ive installed logstash v6.3.2 via yum on CentOS7. When I start it via systemctl start logstash it appears to start properly but then exits almost immediately. There are no (apparent) errors shown. (...
user avatar
  • 2,297
0 votes
1 answer
2k views

Need help increasing performance of logstash

We are in the process of deploying an ELK stack and need advice and general recommendations regarding the performance of the cluster and more specifically, logstash. So the current setup we have now ...
user avatar
3 votes
2 answers
2k views

Windows Event Forwarding large scale with multiple domains

Let me start off explaining what I'm trying to do: We have an RMM tool installed on lots of Windows servers. It is able to send Windows Event Logs to a central store, but not in an efficient or ...
user avatar
2 votes
1 answer
12k views

Filebeat can't connect to logstash on another server

Filebeat (11.11.11.11) can't connect to logstash (22.22.22.22) on another server (connection reset by peer). But filebeat services from other servers can do it. Also I can connect from this server(11....
user avatar
  • 179
1 vote
1 answer
2k views

Retrieve pfSense/freeBSD logs with elk

I am attempting to centralize logs from different systems. I installed the Elastick Stack (Elasticsearch, Logstash, Kibana) and WAZUH OSSEC on one server (named elk). I have installed the OSSEC ...
user avatar
  • 120
0 votes
2 answers
595 views

Logstash isn't send info to elasticsearch

I have a Kibana environemtent installed, but isn't working. When I try to configure kibana, kibana says: Couldn't find any Elasticsearch data When I list indexes of elasticseach http://localhost:...
user avatar
  • 101
1 vote
1 answer
121 views

Local Logstash failing to forward /var/log/syslog to central Logstash

I want my local instance of Logstash to forward syslog and all /var/log/* files to a remote, central instance of Logstash. The pipeline in /etc/logstash/conf/logstash-local.conf looks as follows: ...
user avatar
  • 187
3 votes
1 answer
3k views

Why is this exclude_lines in filebeat excluding all logs?

I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-...
user avatar
0 votes
1 answer
354 views

Transform Apache Cookie logs to json with rsyslogd

I am logging cookie headers, some cookies may have due date, some others not, the same is tru for other cookie parameters, I need to implement a centrilaized logging, so before sending those to ...
user avatar
2 votes
1 answer
200 views

ELK logstash and core grok patterns

I'm evaluating the ELK stack with filebeat & logstash across a diverse range of applications/ servers. I understand the power of customising my own grok patterns for each application/log, but to ...
user avatar
1 vote
1 answer
201 views

logstash: Running update via RPM stepped on my logstash.yml

Using RPM to update from logstash 5.4.1 to 5.6.4 Everything went great, but the command: yum localinstall logstash-5.6.4.rpm resulted in my /etc/logstash/logstash.yml getting overwritten by the ...
user avatar
0 votes
1 answer
1k views

Info sent from Logstash via elastic output not showing in Kibana, but file output works fine - what am I doing wrong?

I have an "ELK stack" configuration and, at first, was doing the standard 'filebeat' syslog feeding from logstash with the elasticsearch output plugin. It worked just fine. Now I have added a TCP ...
user avatar
  • 73
0 votes
1 answer
14k views

Auto delete elasticsearch data older than 30 days

I have setup a ELK stack to collect logs at central server. It is working perfectly. But by default it is holding elasticsearch index/data permanently. We just want to maintain the data for 30Days. ...
user avatar
-1 votes
1 answer
571 views

How to display filebeat logs as a single file in kibana?

I have used ELK for displaying my middleware logs in kibana. for that I am using multiline patterns which mentioned below, after those logs displaying seperatly like separate expandable. How to ...
user avatar
  • 189
2 votes
0 answers
237 views

IIS not UTF-8 encoding logs correctly. Bug? [closed]

My server's log file is set to encode in UTF-8, it does not seem to be encoding the text correctly. For example the cookie section sometimes has a £ sign in. But in the log it is written as 0xA3 not ...
user avatar
  • 213
2 votes
0 answers
3k views

Logstash cannot find logstash.yml although it exists

Testing logstash as instructed here. This is the command I execute (it doesn't work without the --path.settings too): /usr/share/logstash/bin/logstash -f logstash-simple.conf --path.settings /etc/...
user avatar
  • 251
1 vote
1 answer
208 views

Logstash input log stream via port

Is there any way to read logstash raw input data that is forwarded via certain port? input settings are as below input { tcp { port => 1234 type => "test" } } and ...
user avatar
1 vote
1 answer
3k views

Logtash grok / multiline confusion

My real patterns are more complex but I have tried to boil the problem down to the core issue. Something I don't understand. Please try this out on http://grokconstructor.appspot.com/do/match I'm ...
user avatar
  • 308

1
2 3 4 5 6