Questions tagged [logstash]
logstash is a tool for collecting and distributing log events.
260
questions
1
vote
0
answers
15
views
LogStash and parsing OPNSenser logs
My logs are coming in as follows:
<134>May 24 14:39:32 edge.internal filterlog[2535]: 78,,,ffe6d10d1f27a42fc0edc3abb3a6d333,ovpnc1,match,pass,out,4,0x0,,63,61951,0,DF,6,tcp,60,10.8.0.2,20.44.17....
- 3,881
0
votes
0
answers
23
views
Logstash cloudwatch plugin behavior
We have an ELK stack set up on an ec2 instance that stopped working a month ago and I just got it working again (the problem was credentials with the cloudwatch logstash plugin).
What is strange is ...
1
vote
0
answers
44
views
Certificate only works from client side, how do I debug something like this? (graylog/filebeat/JVM keystore)
I created 2 key-certificate pairs with the exact same method. However, while trying to setup TLS on my graylog server to a remote filebeat node, it does not successfully connect when trying to connect ...
- 23
2
votes
0
answers
365
views
rsyslog is not forwarding logs to elasticsearch
I'm trying to configure rsyslog to send logs to logstash and then forward them to elasticsearch.
I have create a config file /etc/rsyslog.d/60-output.conf with the following content:
*.* @localhost:...
- 141
0
votes
1
answer
81
views
Can logstash "pull" data?
I have two servers. Server A is running Elasticsearch and Logstash. Server B is running filebeat and is also the server which contains all the logs I'm trying to analyse.
Server A is behind a firewall,...
- 901
0
votes
1
answer
123
views
How to display the logs of a VM on ELK stack which is running on another VM?
-ELK is running on localhost & successfully getting logs.
-Both VM's are on same server.
-OS on both VM'S is Ubuntu 18-04 LTS.
- 13
0
votes
1
answer
70
views
I used elk+redis+filebeat to build the logging platform, and now after redis is replaced by codis, logstash reports？？？
I used elk+redis+filebeat to build the logging platform, and now after redis is replaced by codis, logstash reports:
# [2020-06-18T11:20:54,146][WARN ][logstash.inputs.redis] Redis connection problem {...
- 1
0
votes
1
answer
1k
views
Logstash RabbitMQ output plugin
I am a new to Logstash and
I want to store nginx messages in RabbitMQ queue like this:
Nginx logs -(input)-> Logstash -(output)-> RabbitMQ
logstash config:
filter {
grok {
match => { ...
0
votes
1
answer
1k
views
Ship Logs from application server to ELK server with beats
I am building log analyzer for production environment. My scenario is, i need to setupelasticsearch,Logstash and kibana on one centos7 server which is going to use ELK server another one is apache ...
- 3
0
votes
0
answers
15
views
Adding context paths to ELK log analytics?
I'm looking to install the ELK stack on my application server by following the instructions here:
I can see you can then access the various ELK services on localhost via certain ports. Is there a way ...
0
votes
1
answer
782
views
Grok filter is not working properly
I have Filebeat-7.1 installed in a Debian server, this Filebeat send data from files in this Debian server to server with Logstash 7.6 , here are the files config
Filebeat.yml:
#=====================...
- 1
0
votes
2
answers
369
views
Kibana and Logstash: When is a field not a field?
I've enabled the grok filter in our logstash configuration in order to parse logs from HAProxy using the %{HAPROXYHTTP} and %{HAPROXYTCP} patterns. This seems to work great and viewing the details for ...
- 43.2k
0
votes
1
answer
43
views
can logstash package individual log types into packages an scp them over to a central node?
I am curious whether logstash only processes logs and streams them at the event/line level or can it package all apache access logs for example and send them over zipped up to a central node?
- 177
2
votes
1
answer
193
views
Repair logstash checkpoint file
We had an OutOfMemoryException in our logstash 5.6.6 which lead to the checkpoint. head file not to be written properly:
ls -l /var/lib/logstash/queue/main/
total 266424
-rw-r--r-- 1 logstash ...
- 75
1
vote
2
answers
679
views
What is a good approach for testing Logstash pipelines?
We have several Logstash(es) pulling data from a queue and then inserting records to Elasticsearch.
Is there any good way of testing pipelines programmatically so we can include it as "unit tests" ...
- 187
0
votes
1
answer
572
views
AWS ElasticSearch service does not provide Kibana users management
First I'm going to explain my current architecture. I have an EC2 instance running with a Logstash daemon, as input an http configuration and as output my AWS ElasticSearch instance.
As I said before,...
- 101
0
votes
0
answers
2k
views
ELK - Logstash not picking up syslog events
I'm setting up a ELK cluster using Centos 8 and version 7.4 of Elasticsearch, Logstash and Kibana.
My issue is with Logstash not picking up the events coming through syslog.
Configuring Logstash to ...
- 267
1
vote
0
answers
151
views
init: logstash main process (19281) terminated with status 1
I run Amazon Linux instance with logstash 6.4.0.
[root@****** dragan]# cat /etc/issue
Amazon Linux AMI release 2017.03
I wanted to upgrade logstash so I uninstalled 6.4.0 and tried to install 7.3.2. ...
- 11
0
votes
1
answer
140
views
Can logstash be used as a kafka forwarder?
I am looking at the plugins for Logstash and I noticed it had both an input and output plugin for kafka. For a single topic, can Logstash be used to forward kafka topic messages to another kafka ...
2
votes
0
answers
293
views
Logstash to aggregate logs into ElasticSearch
I am trying to aggregate linux logs using rsyslog into Logstash/ElasticSearch running in EKS.
filebeat is already running in EKS to aggregate Kubernetes container logs.
I have configured rsyslog ...
- 119
1
vote
1
answer
60
views
Documents from Elasticsearch Cluster appearing twice in Grafana
I set up an Elasticsearch cluster with one dedicated master node, two master-eligible data nodes and one coordinating node.
The number of replicas is set to one.
There are two pipelines in Logstash, ...
- 141
0
votes
1
answer
454
views
logstash and rsyslog sends wrong hostname
I'm setting up a centralised logging system, comprising two VM's.
Logger, which acts as the central syslog collector (using rsyslog) and Logstash processor; and
Searcher, which acts as the ...
0
votes
1
answer
859
views
logstash not able to upload data to elasticsearch even the pipeline started
I am using elasticsearch 7.1.1 and logstash 7.1.1. I am trying to upload a log file to elastic search using grok filter.
Pipeline is getting started, but data is not getting uploaded.
Here is my ...
- 11
0
votes
1
answer
389
views
Monitor Java Heap on a Centos Server via Telegraf/Grafana
How to Monitor Java Heap usage on centos server using Telegraf and Grafana? This monitoring is particularly for Logstash process running on the server. I tried to use JSTAT to get data, but it has ...
- 827
0
votes
1
answer
117
views
Using Elasticsearch or Logstash output?
I have been going through a few tutorials on using beats to send data to elasticsearch.
I noticed that some tutorials prefer to use logstash as the output which then outputs to elasticsearch. Some ...
- 423
0
votes
1
answer
238
views
Parsing or Reformatting Logs before feeding them to Splunk or Elastic Search
I have very complex log messages, that I want to reduce to the most important fields in order to save quota.
The log messages are multiline and there is a lot of redundant information in them. A ...
- 131
1
vote
2
answers
2k
views
Line breaks in PHP's stack trace in nginx error logs disturbing logstash analysis
I am using nginx with PHP-FPM and ELK as log file analysis.
When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file.
...
- 289
2
votes
0
answers
3k
views
Parsing JSON event in Logstash
I have log in following format, it is a plain json with nested fields.
{
"level": "info",
"message": {
"req": {
"headers": {
"host": "localhost:8080",
...
- 21
0
votes
1
answer
217
views
logstash charset error (gelf)
I had some trouble with Logstash and Nginx (all running on Docker).
On my docker-compose file i was trying to enable logging from Nginx container to Logstash:
nginx:
container_name: nginx
image: ...
1
vote
1
answer
3k
views
Logstash's ElasticSearch Input Plugin should be used to output to ElasticSearch
We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch.
This feature has been successfully collected by MongoDB Input Plugin in real time.
However, the ElasticSearch ...
- 15
0
votes
1
answer
1k
views
Logstash Input MongoDB Error “No suitable driver found for jdbc:mongo://localhost:27017/test”
logasth conf "input part"
input {
jdbc {
jdbc_driver_library => "/home/jskang/mongo-java-driver-3.2.2.jar"
jdbc_driver_class => "com.mongodb.MongoClient"
jdbc_connection_string =&...
- 15
0
votes
1
answer
1k
views
Complete information into Postfix logs
I'm succesfully using Postfix on Debian to relay my internal e-mail and I am sending logs from the mail.log file to a Logstash instance, and I am showing them on a Kibana dashboards to look for not ...
- 1,853
1
vote
1
answer
129
views
ELK Logs from application disappearing when grok rules are enabled
We have two instances of an application (same application, different database), lets call them app1 and app2. The route the logs take is:
appX runs filebeat, which picks up the applications log file ...
- 1,647
0
votes
1
answer
204
views
Is there any version of Logstash that can send logs to both Elasticsearch 1.5 and 6.3?
I have an old Elasticsearch v1.5 domain on Amazon AWS, keeping 30 days of logs.
It seems that it's not possible to upgrade it, so i have created a new domain with version ES 6.3.
Now i have Logstash ...
3
votes
0
answers
2k
views
Logstash losing connection to Elasticsearch nodes
We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We ...
- 31
0
votes
1
answer
646
views
logstash won't run as a service
Ive installed logstash v6.3.2 via yum on CentOS7. When I start it via systemctl start logstash it appears to start properly but then exits almost immediately. There are no (apparent) errors shown.
(...
- 2,369
0
votes
1
answer
2k
views
Need help increasing performance of logstash
We are in the process of deploying an ELK stack and need advice and general recommendations regarding the performance of the cluster and more specifically, logstash.
So the current setup we have now ...
- 153
3
votes
2
answers
2k
views
Windows Event Forwarding large scale with multiple domains
Let me start off explaining what I'm trying to do:
We have an RMM tool installed on lots of Windows servers. It is able to send Windows Event Logs to a central store, but not in an efficient or ...
- 33
2
votes
1
answer
13k
views
Filebeat can't connect to logstash on another server
Filebeat (11.11.11.11) can't connect to logstash (22.22.22.22) on another server (connection reset by peer). But filebeat services from other servers can do it.
Also I can connect from this server(11....
- 179
1
vote
1
answer
2k
views
Retrieve pfSense/freeBSD logs with elk
I am attempting to centralize logs from different systems.
I installed the Elastick Stack (Elasticsearch, Logstash, Kibana) and WAZUH OSSEC on one server (named elk).
I have installed the OSSEC ...
- 120
0
votes
2
answers
699
views
Logstash isn't send info to elasticsearch
I have a Kibana environemtent installed, but isn't working.
When I try to configure kibana, kibana says:
Couldn't find any Elasticsearch data
When I list indexes of elasticseach http://localhost:...
- 101
1
vote
1
answer
151
views
Local Logstash failing to forward /var/log/syslog to central Logstash
I want my local instance of Logstash to forward syslog and all /var/log/* files to a remote, central instance of Logstash.
The pipeline in /etc/logstash/conf/logstash-local.conf looks as follows:
...
- 187
3
votes
1
answer
3k
views
Why is this exclude_lines in filebeat excluding all logs?
I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-...
- 1,076
0
votes
1
answer
400
views
Transform Apache Cookie logs to json with rsyslogd
I am logging cookie headers, some cookies may have due date, some others not, the same is tru for other cookie parameters, I need to implement a centrilaized logging, so before sending those to ...
- 115
2
votes
1
answer
240
views
ELK logstash and core grok patterns
I'm evaluating the ELK stack with filebeat & logstash across a diverse range of applications/ servers.
I understand the power of customising my own grok patterns for each application/log, but to ...
- 141
1
vote
1
answer
204
views
logstash: Running update via RPM stepped on my logstash.yml
Using RPM to update from logstash 5.4.1 to 5.6.4
Everything went great, but the command:
yum localinstall logstash-5.6.4.rpm
resulted in my /etc/logstash/logstash.yml getting overwritten by the ...
0
votes
1
answer
1k
views
Info sent from Logstash via elastic output not showing in Kibana, but file output works fine - what am I doing wrong?
I have an "ELK stack" configuration and, at first, was doing the standard 'filebeat' syslog feeding from logstash with the elasticsearch output plugin. It worked just fine.
Now I have added a TCP ...
- 73
1
vote
1
answer
15k
views
Auto delete elasticsearch data older than 30 days
I have setup a ELK stack to collect logs at central server. It is working perfectly. But by default it is holding elasticsearch index/data permanently. We just want to maintain the data for 30Days. ...
- 189
-1
votes
1
answer
610
views
How to display filebeat logs as a single file in kibana?
I have used ELK for displaying my middleware logs in kibana. for that I am using multiline patterns which mentioned below, after those logs displaying seperatly like separate expandable. How to ...
- 209
2
votes
0
answers
316
views
IIS not UTF-8 encoding logs correctly. Bug? [closed]
My server's log file is set to encode in UTF-8, it does not seem to be encoding the text correctly.
For example the cookie section sometimes has a £ sign in. But in the log it is written as 0xA3 not ...
- 213