Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

1
vote
1answer
948 views

vm.max_map_count problems on GKE ElasticSearch StatefulSet

A problem appeared on working ElasticSearch cluster on GKE. Nodes with "data" roles began to crash unexpectedly with an error: max virtual memory areas vm.max_map_count [65530] is too low, increase ...
1
vote
1answer
2k views

Occasional “failed to respond” from AWS hosted Elasticsearch

We have an Elasticsearch cluster hosted on Amazon Elasticsearch Service (AWS). We are using the Jest Java HTTP Rest client for ElasticSearch. Every now and then (perhaps 1 in 10,000 requests), it ...
0
votes
1answer
99 views

Elasticsearch - Kubernetes [handshake failed, mismatched cluster name]

When setting up Elasticsearch cluster on Kubernetes with 2x data, 2x master and 2x client nodes (each container on sepearete physical node) i get the following error in client-node: [2019-01-28T12:25:...
1
vote
1answer
53 views

Connect MongoDB to ElasticSearch by MongoConnector in Kubernetes

I have a MongoDB and ElasticSearch cluster which I've run by Kubernetes. I am trying to connect my ElasticSearch to MongoDB on Kubernetes but I can't find any relevant document. Can anyone help me ...
0
votes
0answers
16 views

Elastic search monitoring with graphite grafana (Better solution?)

We have created a grafana dashboard which is configured for Elasticsearch resource monitoring and we are using python script to fetch resource information from ES and put it in to Graphite. We are ...
1
vote
1answer
956 views

iptables to allow input and output traffic to and from web server only

I have an Elastic Search server which seems to have been exploited (it's being used for a DDoS attack having had NO firewall for about a month). As a temporary measure while I create a new one I was ...
0
votes
0answers
9 views

Elasticsearch snapshot to azure - option not appearing

I've installed the Azure repository plugin for 6.7, and configured a blob for it to write to. The directions at https://www.elastic.co/guide/en/elasticsearch/plugins/6.7/repository-azure.html have ...
1
vote
1answer
23 views

Cannot connect to open port in remote server

On my remote server, I installed ElasticSearch, which can be accessed at http://localhost:9200/: I defined an inbound security rule, Port9200, in the firewall for my remote server: As you can see, ...
1
vote
1answer
1k views

Signing/Generating SSL Certificates for ElasticSearch (search-guard)

I am trying to build my own ElasticSearch cluster and I decided to use an open source plugin called Search Guard. It requires generation of client and host/node certificates. In its demo there is a ...
1
vote
0answers
50 views

Parsing JSON event in Logstash

I have log in following format, it is a plain json with nested fields. { "level": "info", "message": { "req": { "headers": { "host": "localhost:8080", ...
9
votes
1answer
418 views

Elasticsearch dies when Logstash attempts to write data

I've got a Raspberry Pi 2 (latest Raspbian as of Apr 2015) setup that last week was running both ElasticSearch and Logstash on a test network (not a straightforward setup, but it was stable for over a ...
3
votes
3answers
2k views

kibana: no indices error

None of the existing answers helped, so here a new question. Usecase: Redirecting syslog (or) monitoring static file. I have successfully installed logstash (1.4.2), elasticsearch(1.1.1) and kibana(...
0
votes
0answers
37 views

Getting Apache2 combined logs into the Elastic Stack and using dashboards

I am looking into the ELK stack, and I mostly got it running. My goal would be to get the Apache2 dashboards running in Kibana, but I am unable to. This is how far I got: **On the Apache side:* I am ...
1
vote
0answers
29 views

ElasticSearch not starting properly

On startup, our ElasticSearch server throws aJava exception I can see in the elasticsearch.log: [2019-02-27T15:51:59,908][WARN ][o.e.m.j.JvmGcMonitorService] [KibjtLy] [gc][young][263][19] duration [...
1
vote
1answer
122 views

How to increase disk size in a stateful set

I'm managing an Elasticsearch deployment in Kubernetes. I see that the disk storage is getting close to full, so I would like to increase the size of the persistent volumes. I want to change this ...
0
votes
0answers
28 views

Moving data to a new elasticsearch index with more shards

We seem to be running out of shards on some of our larger indices on Elasticsearch 1.7. In order to utilize larger clusters with more nodes, say 20 instead of 10, I'm hoping that we can create new ...
0
votes
1answer
1k views

Why doesn't logstash grab or index the files from the mapped drive?

I don't understand why logstash is so finicky with network resources. I shared a folder on another machine and then mapped it as Z: under Windows Explorer. I've verified the path and everything. I can ...
0
votes
3answers
2k views

nginx serve Elasticsearch and Kibana on the same host

Totally new to nginx, I need a conf file to make nginx act as a reverse proxy to serve request on the same host to elasticsearch and kibana with different url path. I mean, I want: localhost/es -> ...
0
votes
0answers
86 views

Logstash: Failed to decode CEF payload

I am parsing via the cef input plugin in Logstash log files in cef format to elasticsearch. At some log files I recevice the message Failed to decode CEF payload. It seems that the problem could be ...
0
votes
1answer
81 views

Why we need to go for elasticsearch cluster with kubernetes?

I have installed kubernetes on my DC and all is working fine. I have tested several services like mongodb and web services. But my goal is to find a good solution for Elasticsearch big data. From ...
1
vote
0answers
31 views

VPC subnet error when deploying Bitname Elasticsearch to non-default GCE network

Deploying Bitnami's single-VM Elasticsearch. The error is: Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network It works fine if ...
0
votes
0answers
35 views

Filebeat is not sending data to Logstash

I’m trying to send alerts from Snort IDS to Elasticsearch, therefore I'm using 3 technologies: Elasticsearch- https://pastebin.com/uCNMaZFJ Logstash-https://pastebin.com/zgnbbw9K Filebeat- https://...
0
votes
0answers
77 views

fluentd error parsing auth.log

I have fluentd setup for Kubernetes which is working just fine but when add more sources it fails. I'm trying to get auth.log like this: <source> @id authlog @type tail format none ...
0
votes
2answers
87 views

Elastic Search using a lot of memory despite low heap size

We have configured Elastic Search JVM options /etc/elasticsearch/jvm.options to use a fixed heap size of -Xms512m -Xmx512m This can be confirmed as working when checking the Elastic Search process: ...
0
votes
1answer
1k views

Error while setting up grafana with graphite and nginx

While trying to setup grafana with graphite using nginx server. I am getting this error: No 'Access-Control-Allow-Origin' header is present on the requested resource. I tried changing configuration ...
0
votes
1answer
2k views

elasticsearch: max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536]

When I tried to run the logging aggregation I found out the following error generated by elasticsearch: [2018-02-04T13:44:04,259][INFO ][o.e.b.BootstrapChecks ] [elasticsearch-logging-0] bound or ...
0
votes
0answers
17 views

Migrating from a public elastic cluster to VPC cluster

I currently, have one public elastic cluster with one data node(m3.xlarge.elasticsearch) which is working fine. I would like to move from the public domain to VPC based domain. I was able to take a ...
0
votes
1answer
1k views

Kibana @timestamp mapping & filter

I'm using following system/package: $ cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) $ rpm -q filebeat filebeat-1.3.0-1.x86_64 $ with /etc/filebeat/filebeat.yml: $ cat /etc/...
1
vote
1answer
2k views

cannot validate certificate - doesn't contain any IP SAN

I am currently in the process of installing ELK ( ElastricSearch, LogStash & Kibana) stack. My ELK server IP address is 172.29.225.32. Elastic Search config is :: # -----------------------------...
0
votes
2answers
2k views

What are records in 'netstat' command are when using logstash?

I use logstash with embedded elasticsearch. My logstash config looks like the following: output { tdout { debug => true debug_format => "json"} elasticsearch { embedded => true ...
0
votes
0answers
95 views

Logstash isn't accepting the self signed certificates and throws a sslv3 alert handshake failure

I have set up an elsatic stack in my laptop and I have configured full stack (filebeat, logstash, elasticsearch and kibana) in it. But now I need to connect filebeat and logstash in a secure manner ...
0
votes
0answers
30 views

AWS Elasticsearch access policy with Kinesis and EC2

I am currently working on an AWS setup where data is pushed to an Elasticsearch service via Amazon Kinesis (Firehose). This data has to be accessible from all EC2 instances that are connected to a ...
4
votes
2answers
2k views

Elasticsearch performance tuning

In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge. Logstash conf ...
5
votes
2answers
3k views

“Index Patterns: Please specify a default index pattern” in Kibana

I'm trying to create a simple hello world for ELK and be able to see kibana reports via the internet. I've installed kibana, logstash, nginx and elastic search. Here's my /etc/logstash/conf.d/10-...
0
votes
1answer
4k views

Elastic Search Unassigned Shards

my cluster state was green and after restart of service one shard remains UNASSIGNED and status goes to yellow. I have 2 machines with 5 shard and 1 replica set settings. I am using default config ...
2
votes
3answers
2k views

Logstash / Elasticsearch - trasform fields to lowercase in output conf

I have a standard ELK stack currently storing numerous log outputs. I'm trying to separate my indices to be source-specific. As part of my FileBeats config, some standard fields are always generated ...
1
vote
1answer
70 views

Logstash's ElasticSearch Input Plugin should be used to output to ElasticSearch

We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch. This feature has been successfully collected by MongoDB Input Plugin in real time. However, the ElasticSearch ...
0
votes
1answer
173 views

Logstash Input MongoDB Error “No suitable driver found for jdbc:mongo://localhost:27017/test”

logasth conf "input part" input { jdbc { jdbc_driver_library => "/home/jskang/mongo-java-driver-3.2.2.jar" jdbc_driver_class => "com.mongodb.MongoClient" jdbc_connection_string =&...
0
votes
0answers
159 views

Ceph vs GlusterFS for elasticsearch storage

Elasticsearch will be holding logs mostly, so we expect sequential read/writes. Does anyone have experience on this? Elasticsearch will run via openshift, if it makes any difference.
0
votes
1answer
74 views

Configure Kibana web page (only) to require a login

I have Elasticsearch 6.4.3 and Kibana 6.4.3 installed on Debian 8 Jessie. I've looked into X-Pack and Shield, which both seem to be mostly concerned with security within Elasticsearch, and how to ...
2
votes
0answers
43 views

AWS ElasticSearch on private subnet - safe to have no signed url/access policy?

I have an ES cluster setup within our VPC and attached to one of our private subnets. It's also attached to a security group allowing access from particular instances. I have the following policy ...
0
votes
0answers
120 views

Multiple multiline parsers in Kubernetes Filebeat config

I'm using Filebeat to ship logs to Kubernetes. How can I configure multiline collation differently based on different conditions? E.g., use a Java log regex for my Java containers, and a PHP regex for ...
0
votes
1answer
150 views

Can't find docker log files for Filebeat

I'm trying to aggregate logs from my Kubernetes cluster into Elasticsearch server. To do that, I've deployed Filebeat on the cluster, but I think it doesn't have a chance to work since in the /var/lib/...
4
votes
1answer
135 views

elasticsearch server is unreachable every 2 hours

This question is related to this one. We now know that the errors come from elasticsearch. The problems are still not resolved despite the modifications and optimizations made on the es instance. ...
6
votes
3answers
6k views

Elastic Search listening only on IPv6 [CentOS]

I've installed Elastic Search 1.7.4 on CentOS 7.2 this way: wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.4.noarch.rpm sudo rpm -ivh elasticsearch-1.7.4.noarch.rpm ...
1
vote
1answer
31 views

ELK Logs from application disappearing when grok rules are enabled

We have two instances of an application (same application, different database), lets call them app1 and app2. The route the logs take is: appX runs filebeat, which picks up the applications log file ...
0
votes
2answers
409 views

ELK stack on AWS and web app on Digital Ocean, how and can I grab logs from the later?

I have a python website hosted on digital ocean and ELK stack - Elastic Search, Kibana, Logstash - on AWS. I'm trying to understand how to feed the logs files from my python web app on digital ocean ...
0
votes
1answer
82 views

Is there any version of Logstash that can send logs to both Elasticsearch 1.5 and 6.3?

I have an old Elasticsearch v1.5 domain on Amazon AWS, keeping 30 days of logs. It seems that it's not possible to upgrade it, so i have created a new domain with version ES 6.3. Now i have Logstash ...
0
votes
0answers
231 views

Running Sonarqube, elasticsearch throws error in sonar.log

Installed and started Sonarqube up, and in the sonar.log it complains about access denied for elasticsearch.yml. It's surely permissions, but I can't figure out what exactly. Sonarqube 7.3 and ...
0
votes
0answers
23 views

fluent and parsing of logs

I'm trying to centralize logs using fluent. That part seems to be ok, I have a central fluent and other nodes and I use forward plugin to push them. Once on the central fluent I have 2 outputs, one ...