Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

1
vote
1answer
17 views

Cannot connect to open port in remote server

On my remote server, I installed ElasticSearch, which can be accessed at http://localhost:9200/: I defined an inbound security rule, Port9200, in the firewall for my remote server: As you can see, ...
1
vote
0answers
26 views

Connect MongoDB to ElasticSearch by MongoConnector in Kubernetes

I have a MongoDB and ElasticSearch cluster which I've run by Kubernetes . I am trying to connect my ElasticSearch to MongoDB on Kubernetes but I can't find any relevant document. Can anyone help me ...
1
vote
0answers
20 views

Parsing JSON event in Logstash

I have log in following format, it is a plain json with nested fields. { "level": "info", "message": { "req": { "headers": { "host": "localhost:8080", ...
0
votes
0answers
17 views

Getting Apache2 combined logs into the Elastic Stack and using dashboards

I am looking into the ELK stack, and I mostly got it running. My goal would be to get the Apache2 dashboards running in Kibana, but I am unable to. This is how far I got: **On the Apache side:* I am ...
1
vote
0answers
19 views

ElasticSearch not starting properly

On startup, our ElasticSearch server throws aJava exception I can see in the elasticsearch.log: [2019-02-27T15:51:59,908][WARN ][o.e.m.j.JvmGcMonitorService] [KibjtLy] [gc][young][263][19] duration [...
1
vote
1answer
59 views

How to increase disk size in a stateful set

I'm managing an Elasticsearch deployment in Kubernetes. I see that the disk storage is getting close to full, so I would like to increase the size of the persistent volumes. I want to change this ...
0
votes
0answers
19 views

Moving data to a new elasticsearch index with more shards

We seem to be running out of shards on some of our larger indices on Elasticsearch 1.7. In order to utilize larger clusters with more nodes, say 20 instead of 10, I'm hoping that we can create new ...
0
votes
0answers
47 views

Logstash: Failed to decode CEF payload

I am parsing via the cef input plugin in Logstash log files in cef format to elasticsearch. At some log files I recevice the message Failed to decode CEF payload. It seems that the problem could be ...
0
votes
1answer
75 views

Why we need to go for elasticsearch cluster with kubernetes?

I have installed kubernetes on my DC and all is working fine. I have tested several services like mongodb and web services. But my goal is to find a good solution for Elasticsearch big data. From ...
1
vote
0answers
31 views

VPC subnet error when deploying Bitname Elasticsearch to non-default GCE network

Deploying Bitnami's single-VM Elasticsearch. The error is: Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network It works fine if ...
0
votes
0answers
32 views

Filebeat is not sending data to Logstash

I’m trying to send alerts from Snort IDS to Elasticsearch, therefore I'm using 3 technologies: Elasticsearch- https://pastebin.com/uCNMaZFJ Logstash-https://pastebin.com/zgnbbw9K Filebeat- https://...
0
votes
0answers
40 views

fluentd error parsing auth.log

I have fluentd setup for Kubernetes which is working just fine but when add more sources it fails. I'm trying to get auth.log like this: <source> @id authlog @type tail format none ...
0
votes
0answers
46 views

Elasticsearch - Kubernetes [handshake failed, mismatched cluster name]

When setting up Elasticsearch cluster on Kubernetes with 2x data, 2x master and 2x client nodes (each container on sepearete physical node) i get the following error in client-node: [2019-01-28T12:25:...
0
votes
0answers
17 views

Migrating from a public elastic cluster to VPC cluster

I currently, have one public elastic cluster with one data node(m3.xlarge.elasticsearch) which is working fine. I would like to move from the public domain to VPC based domain. I was able to take a ...
0
votes
2answers
53 views

Elastic Search using a lot of memory despite low heap size

We have configured Elastic Search JVM options /etc/elasticsearch/jvm.options to use a fixed heap size of -Xms512m -Xmx512m This can be confirmed as working when checking the Elastic Search process: ...
0
votes
0answers
55 views

Logstash isn't accepting the self signed certificates and throws a sslv3 alert handshake failure

I have set up an elsatic stack in my laptop and I have configured full stack (filebeat, logstash, elasticsearch and kibana) in it. But now I need to connect filebeat and logstash in a secure manner ...
0
votes
0answers
21 views

AWS Elasticsearch access policy with Kinesis and EC2

I am currently working on an AWS setup where data is pushed to an Elasticsearch service via Amazon Kinesis (Firehose). This data has to be accessible from all EC2 instances that are connected to a ...
1
vote
1answer
39 views

Logstash's ElasticSearch Input Plugin should be used to output to ElasticSearch

We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch. This feature has been successfully collected by MongoDB Input Plugin in real time. However, the ElasticSearch ...
0
votes
1answer
139 views

Logstash Input MongoDB Error “No suitable driver found for jdbc:mongo://localhost:27017/test”

logasth conf "input part" input { jdbc { jdbc_driver_library => "/home/jskang/mongo-java-driver-3.2.2.jar" jdbc_driver_class => "com.mongodb.MongoClient" jdbc_connection_string =&...
0
votes
0answers
140 views

Ceph vs GlusterFS for elasticsearch storage

Elasticsearch will be holding logs mostly, so we expect sequential read/writes. Does anyone have experience on this? Elasticsearch will run via openshift, if it makes any difference.
0
votes
1answer
58 views

Configure Kibana web page (only) to require a login

I have Elasticsearch 6.4.3 and Kibana 6.4.3 installed on Debian 8 Jessie. I've looked into X-Pack and Shield, which both seem to be mostly concerned with security within Elasticsearch, and how to ...
2
votes
0answers
37 views

AWS ElasticSearch on private subnet - safe to have no signed url/access policy?

I have an ES cluster setup within our VPC and attached to one of our private subnets. It's also attached to a security group allowing access from particular instances. I have the following policy ...
0
votes
0answers
112 views

Multiple multiline parsers in Kubernetes Filebeat config

I'm using Filebeat to ship logs to Kubernetes. How can I configure multiline collation differently based on different conditions? E.g., use a Java log regex for my Java containers, and a PHP regex for ...
0
votes
1answer
127 views

Can't find docker log files for Filebeat

I'm trying to aggregate logs from my Kubernetes cluster into Elasticsearch server. To do that, I've deployed Filebeat on the cluster, but I think it doesn't have a chance to work since in the /var/lib/...
1
vote
1answer
30 views

ELK Logs from application disappearing when grok rules are enabled

We have two instances of an application (same application, different database), lets call them app1 and app2. The route the logs take is: appX runs filebeat, which picks up the applications log file ...
0
votes
0answers
57 views

Logstash monitoring: Verify status of pipelines

Trying to establish monitoring for http://elasticsearchnodeexample.com:9600/_node/stats/pipelines to verify status of 2 pipelines Output : "pipelines": { "pipeline1": { "events": { ...
0
votes
0answers
186 views

Running Sonarqube, elasticsearch throws error in sonar.log

Installed and started Sonarqube up, and in the sonar.log it complains about access denied for elasticsearch.yml. It's surely permissions, but I can't figure out what exactly. Sonarqube 7.3 and ...
0
votes
1answer
71 views

Is there any version of Logstash that can send logs to both Elasticsearch 1.5 and 6.3?

I have an old Elasticsearch v1.5 domain on Amazon AWS, keeping 30 days of logs. It seems that it's not possible to upgrade it, so i have created a new domain with version ES 6.3. Now i have Logstash ...
0
votes
0answers
22 views

fluent and parsing of logs

I'm trying to centralize logs using fluent. That part seems to be ok, I have a central fluent and other nodes and I use forward plugin to push them. Once on the central fluent I have 2 outputs, one ...
1
vote
0answers
197 views

Logstash losing connection to Elasticsearch nodes

We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We ...
0
votes
1answer
25 views

Elasticsearch on Google Compute - Network issues

I just moved my cluster from AWS EC2 to Google Compute, and looking at the logs it seems there are constant network issues. This happens with 2 specific nodes, several times a day. It starts with ...
0
votes
0answers
35 views

ElasticSearch make replica shard primary

We have a small ElasticSearch 2.4 cluster containing 2 Data Nodes and multiple Indicies. The indicies have 2 primary shards and 1 replica each. If we create a new index using our default template, ...
0
votes
1answer
768 views

vm.max_map_count problems on GKE ElasticSearch StatefulSet

A problem appeared on working ElasticSearch cluster on GKE. Nodes with "data" roles began to crash unexpectedly with an error: max virtual memory areas vm.max_map_count [65530] is too low, increase ...
0
votes
0answers
12 views

Can the latest version of Fluentd, only work with ES-6, or also with ElasticSearch-2?

Can the latest version of Fluentd, only work with ES-6, or also with ElasticSearch-2? Background: We want to know if we can temporarily build a new ELK-6 stack, update fluentd and have updated fluend ...
0
votes
1answer
287 views

Simplest upgrade data from ElasticSearch 2 to ElasticSearch 6?

We're looking to migrate our clusters with ElasticSearch 2.x to the latest version. What is the simplest way to do this? I already wrote a deploy script for deploying the ELK 2 stack and the ELK 6 ...
0
votes
1answer
694 views

memory locking requested for elasticsearch process but memory is not locked

ElasticSearch 5.6.7 throws somewhat misleading following error and refuse to start: [2018-08-02T10:07:50,333][ERROR][o.e.b.Bootstrap ] [esm56n3] node validation exception [2] bootstrap ...
0
votes
0answers
63 views

Unable to update ElasticSearch access policy using AWS CLI

I have an access policy that looks something like this: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "...
1
vote
1answer
6k views

What does “GC (Allocation Failure)” mean in my ElasticSearch 5.6 logs?

I see these constantly in my Elasticsearch 5.6.3 logs. Is this a signal that I should scale up and add more RAM? Or is this just normal operations for ElasticSearch? [GC (Allocation Failure) [...
0
votes
1answer
148 views

esrally.exceptions.SystemSetupError: ('JAVA_HOME points to JDK 8 but it should point to JDK 10.', None) when running a race on esrally

The Dockerfile of my docker 17.03.1-ce is as follows FROM ubuntu:16.04 # Specifying versions is better?? RUN apt-get update && \ apt-get install -y \ python3 \ ...
0
votes
3answers
196 views

Restart Elasticsearch cluster when adding new node

I have an ELK stack for our logging needs. The stack is 1 Logstash server, 3 Elasticsearch, and 1 Kibana server. All have playbooks in Ansible to redeploy when needed. When I go to redeploy the ...
0
votes
3answers
1k views

nginx serve Elasticsearch and Kibana on the same host

Totally new to nginx, I need a conf file to make nginx act as a reverse proxy to serve request on the same host to elasticsearch and kibana with different url path. I mean, I want: localhost/es -> ...
0
votes
1answer
250 views

Need help increasing performance of logstash

We are in the process of deploying an ELK stack and need advice and general recommendations regarding the performance of the cluster and more specifically, logstash. So the current setup we have now ...
0
votes
0answers
16 views

Load ElasticSearch index (?) on Redis to eliminate downtime

I’ve been assigned to do some weird task to my opinion. Some background: Project’s infrastructure comprises by Elasticsearch cluster and an API. The API hits elasticsearch for data (plain and simple)....
1
vote
0answers
182 views

Configure Google Cloud Storage plugin with Curator cronjob on GKE Elasticsearch cluster

I deployed an Elasticsearch cluster on GKE, using this project as a starting point: https://github.com/pires/kubernetes-elasticsearch-cluster I would like to configure two Curator jobs by using ...
0
votes
1answer
765 views

Can't see pod logs in Stackdriver UI for cluster deployed on GKE

I deployed an Elasticsearch cluster on Google Kubernetes Engine. I am able to view logs of specific pods with the 'kubectl logs' command from my PC. >kubectl logs es-data-0 ... [2018-06-18T07:35:...
1
vote
1answer
424 views

How to move/migrate indices from self hosted Elasticsearch to AWS Elasticsearch service (cloud)

I have a self hosted Elasticsearch 6.2 cluster (2 master nodes, ~200Gb data each). I plan to move to AWS Elasticsearch service & it's not possible to ssh into it. What's the fastest way to move ...
0
votes
1answer
28 views

ElasticSearch including or vs database?

I am trying to understand what ElasticSearch exactly does when it comes to persistence. When I index documents in ElasticSearch, I also within this step save them within ElasticSearch. Though the ...
0
votes
0answers
7 views

How does unicast work in ElasticSearch

Regards to this document: https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html#unicast.hosts It stated that, all node should specify this configuration of all the ...
0
votes
0answers
138 views

Elasticsearch docker containers in a swarm cannot connect to master until they fail and restart

I am trying to setup an ELK stack in a swarm. Just to start I have one of each node type. A master, client, ingest, and data. The master node launches but each of the other nodes cannot find the ...
1
vote
1answer
2k views

Filebeat can't connect to logstash on another server

Filebeat (11.11.11.11) can't connect to logstash (22.22.22.22) on another server (connection reset by peer). But filebeat services from other servers can do it. Also I can connect from this server(11....