Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

0
votes
2answers
165 views

Elastic Search using a lot of memory despite low heap size

We have configured Elastic Search JVM options /etc/elasticsearch/jvm.options to use a fixed heap size of -Xms512m -Xmx512m This can be confirmed as working when checking the Elastic Search process: ...
0
votes
0answers
19 views

Migrating from a public elastic cluster to VPC cluster

I currently, have one public elastic cluster with one data node(m3.xlarge.elasticsearch) which is working fine. I would like to move from the public domain to VPC based domain. I was able to take a ...
0
votes
0answers
161 views

Logstash isn't accepting the self signed certificates and throws a sslv3 alert handshake failure

I have set up an elsatic stack in my laptop and I have configured full stack (filebeat, logstash, elasticsearch and kibana) in it. But now I need to connect filebeat and logstash in a secure manner ...
0
votes
0answers
40 views

AWS Elasticsearch access policy with Kinesis and EC2

I am currently working on an AWS setup where data is pushed to an Elasticsearch service via Amazon Kinesis (Firehose). This data has to be accessible from all EC2 instances that are connected to a ...
2
votes
3answers
3k views

Logstash / Elasticsearch - trasform fields to lowercase in output conf

I have a standard ELK stack currently storing numerous log outputs. I'm trying to separate my indices to be source-specific. As part of my FileBeats config, some standard fields are always generated ...
1
vote
1answer
174 views

Logstash's ElasticSearch Input Plugin should be used to output to ElasticSearch

We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch. This feature has been successfully collected by MongoDB Input Plugin in real time. However, the ElasticSearch ...
0
votes
1answer
225 views

Logstash Input MongoDB Error “No suitable driver found for jdbc:mongo://localhost:27017/test”

logasth conf "input part" input { jdbc { jdbc_driver_library => "/home/jskang/mongo-java-driver-3.2.2.jar" jdbc_driver_class => "com.mongodb.MongoClient" jdbc_connection_string =&...
0
votes
0answers
200 views

Ceph vs GlusterFS for elasticsearch storage

Elasticsearch will be holding logs mostly, so we expect sequential read/writes. Does anyone have experience on this? Elasticsearch will run via openshift, if it makes any difference.
0
votes
1answer
113 views

Configure Kibana web page (only) to require a login

I have Elasticsearch 6.4.3 and Kibana 6.4.3 installed on Debian 8 Jessie. I've looked into X-Pack and Shield, which both seem to be mostly concerned with security within Elasticsearch, and how to ...
2
votes
0answers
56 views

AWS ElasticSearch on private subnet - safe to have no signed url/access policy?

I have an ES cluster setup within our VPC and attached to one of our private subnets. It's also attached to a security group allowing access from particular instances. I have the following policy ...
0
votes
0answers
134 views

Multiple multiline parsers in Kubernetes Filebeat config

I'm using Filebeat to ship logs to Kubernetes. How can I configure multiline collation differently based on different conditions? E.g., use a Java log regex for my Java containers, and a PHP regex for ...
0
votes
1answer
178 views

Can't find docker log files for Filebeat

I'm trying to aggregate logs from my Kubernetes cluster into Elasticsearch server. To do that, I've deployed Filebeat on the cluster, but I think it doesn't have a chance to work since in the /var/lib/...
4
votes
1answer
136 views

elasticsearch server is unreachable every 2 hours

This question is related to this one. We now know that the errors come from elasticsearch. The problems are still not resolved despite the modifications and optimizations made on the es instance. ...
6
votes
3answers
6k views

Elastic Search listening only on IPv6 [CentOS]

I've installed Elastic Search 1.7.4 on CentOS 7.2 this way: wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.4.noarch.rpm sudo rpm -ivh elasticsearch-1.7.4.noarch.rpm ...
1
vote
1answer
32 views

ELK Logs from application disappearing when grok rules are enabled

We have two instances of an application (same application, different database), lets call them app1 and app2. The route the logs take is: appX runs filebeat, which picks up the applications log file ...
0
votes
2answers
412 views

ELK stack on AWS and web app on Digital Ocean, how and can I grab logs from the later?

I have a python website hosted on digital ocean and ELK stack - Elastic Search, Kibana, Logstash - on AWS. I'm trying to understand how to feed the logs files from my python web app on digital ocean ...
0
votes
1answer
99 views

Is there any version of Logstash that can send logs to both Elasticsearch 1.5 and 6.3?

I have an old Elasticsearch v1.5 domain on Amazon AWS, keeping 30 days of logs. It seems that it's not possible to upgrade it, so i have created a new domain with version ES 6.3. Now i have Logstash ...
0
votes
0answers
311 views

Running Sonarqube, elasticsearch throws error in sonar.log

Installed and started Sonarqube up, and in the sonar.log it complains about access denied for elasticsearch.yml. It's surely permissions, but I can't figure out what exactly. Sonarqube 7.3 and ...
0
votes
0answers
28 views

fluent and parsing of logs

I'm trying to centralize logs using fluent. That part seems to be ok, I have a central fluent and other nodes and I use forward plugin to push them. Once on the central fluent I have 2 outputs, one ...
0
votes
1answer
2k views

“mapper_parsing_exception” error from Elasticsearch failing to parse [timestamp] from %{COMBINEDAPACHELOG}

I've configured Logstash to filter httpd_access_log messages and grok the fields associated with COMBINEDAPACHELOG. However, I'm receiving errors like the following: [2017-02-10T15:37:39,361][WARN ][...
0
votes
1answer
26 views

Elasticsearch on Google Compute - Network issues

I just moved my cluster from AWS EC2 to Google Compute, and looking at the logs it seems there are constant network issues. This happens with 2 specific nodes, several times a day. It starts with ...
1
vote
0answers
271 views

Logstash losing connection to Elasticsearch nodes

We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We ...
0
votes
0answers
52 views

ElasticSearch make replica shard primary

We have a small ElasticSearch 2.4 cluster containing 2 Data Nodes and multiple Indicies. The indicies have 2 primary shards and 1 replica each. If we create a new index using our default template, ...
0
votes
1answer
177 views

esrally.exceptions.SystemSetupError: ('JAVA_HOME points to JDK 8 but it should point to JDK 10.', None) when running a race on esrally

The Dockerfile of my docker 17.03.1-ce is as follows FROM ubuntu:16.04 # Specifying versions is better?? RUN apt-get update && \ apt-get install -y \ python3 \ ...
0
votes
0answers
12 views

Can the latest version of Fluentd, only work with ES-6, or also with ElasticSearch-2?

Can the latest version of Fluentd, only work with ES-6, or also with ElasticSearch-2? Background: We want to know if we can temporarily build a new ELK-6 stack, update fluentd and have updated fluend ...
0
votes
1answer
409 views

Need help increasing performance of logstash

We are in the process of deploying an ELK stack and need advice and general recommendations regarding the performance of the cluster and more specifically, logstash. So the current setup we have now ...
0
votes
1answer
546 views

Simplest upgrade data from ElasticSearch 2 to ElasticSearch 6?

We're looking to migrate our clusters with ElasticSearch 2.x to the latest version. What is the simplest way to do this? I already wrote a deploy script for deploying the ELK 2 stack and the ELK 6 ...
0
votes
1answer
946 views

memory locking requested for elasticsearch process but memory is not locked

ElasticSearch 5.6.7 throws somewhat misleading following error and refuse to start: [2018-08-02T10:07:50,333][ERROR][o.e.b.Bootstrap ] [esm56n3] node validation exception [2] bootstrap ...
-1
votes
2answers
8k views

java.io.IOException: Too many open files on tomcat server

I have web application which is based on JAVA (GWT Framework). I hosted my web application on Redhat Linux Server 6.0 using Tomcat 8.0.9 in front of apache http server connected using AJP connector ...
0
votes
0answers
80 views

Unable to update ElasticSearch access policy using AWS CLI

I have an access policy that looks something like this: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "...
1
vote
1answer
10k views

What does “GC (Allocation Failure)” mean in my ElasticSearch 5.6 logs?

I see these constantly in my Elasticsearch 5.6.3 logs. Is this a signal that I should scale up and add more RAM? Or is this just normal operations for ElasticSearch? [GC (Allocation Failure) [...
0
votes
3answers
302 views

Restart Elasticsearch cluster when adding new node

I have an ELK stack for our logging needs. The stack is 1 Logstash server, 3 Elasticsearch, and 1 Kibana server. All have playbooks in Ansible to redeploy when needed. When I go to redeploy the ...
0
votes
0answers
18 views

Load ElasticSearch index (?) on Redis to eliminate downtime

I’ve been assigned to do some weird task to my opinion. Some background: Project’s infrastructure comprises by Elasticsearch cluster and an API. The API hits elasticsearch for data (plain and simple)....
1
vote
0answers
229 views

Configure Google Cloud Storage plugin with Curator cronjob on GKE Elasticsearch cluster

I deployed an Elasticsearch cluster on GKE, using this project as a starting point: https://github.com/pires/kubernetes-elasticsearch-cluster I would like to configure two Curator jobs by using ...
0
votes
1answer
500 views

How to take Elasticsearch single index backup

Can someone guide me to take single index backup in elasticsearch. When I search for this, I get all commands to take full snapshot.
2
votes
1answer
6k views

How do I feed Elasticsearch my Snort alert log?

I started off yesterday with an ELK howto and got ELK up and running rather easily. Next thing I wanted to do was plug my Snort alert log into it. I configured Logstash (shown below) with a filter and ...
1
vote
1answer
549 views

How to move/migrate indices from self hosted Elasticsearch to AWS Elasticsearch service (cloud)

I have a self hosted Elasticsearch 6.2 cluster (2 master nodes, ~200Gb data each). I plan to move to AWS Elasticsearch service & it's not possible to ssh into it. What's the fastest way to move ...
0
votes
1answer
3k views

elasticsearch nodes not joining cluster

i have three nodes which should autoconfig into a cluster but for some reason they dont seem to want to cluster up. The 3 Hyper-V VM Servers are CENTOS 7 installed as compute with just the tool ...
0
votes
2answers
363 views

Logstash isn't send info to elasticsearch

I have a Kibana environemtent installed, but isn't working. When I try to configure kibana, kibana says: Couldn't find any Elasticsearch data When I list indexes of elasticseach http://localhost:...
0
votes
1answer
28 views

ElasticSearch including or vs database?

I am trying to understand what ElasticSearch exactly does when it comes to persistence. When I index documents in ElasticSearch, I also within this step save them within ElasticSearch. Though the ...
1
vote
1answer
3k views

Filebeat can't connect to logstash on another server

Filebeat (11.11.11.11) can't connect to logstash (22.22.22.22) on another server (connection reset by peer). But filebeat services from other servers can do it. Also I can connect from this server(11....
1
vote
2answers
510 views

elasticsearch Active: failed (Result: exit-code)

[root@66-226-73-146 elasticsearch]# sudo service elasticsearch restart Restarting elasticsearch (via systemctl): [ OK ] [root@66-226-73-146 elasticsearch]# sudo service ...
1
vote
0answers
41 views

CloudWatch Logs, Elasticsearch service expiration

I'm using CloudWatch Logs streaming to Amazon Elasticsearch service (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html) Currently, I've set Events expiration in Cloudwatch ...
1
vote
1answer
51 views

Local Logstash failing to forward /var/log/syslog to central Logstash

I want my local instance of Logstash to forward syslog and all /var/log/* files to a remote, central instance of Logstash. The pipeline in /etc/logstash/conf/logstash-local.conf looks as follows: ...
0
votes
1answer
37 views

Import Elasticsearch 2.X data to 6.X?

I've got some data/logs from Elasticsearch 2.4 & I've got a new Elasticsearch 6.X running. What's the best/correct way to import data from 2.X to 6.x? Thanks!
2
votes
2answers
3k views

Kibana 4, Logstash dashboard: how do I require Nginx authentication when saving but allow anonymous views?

I would like to require auth_basic nginx authentication to save all kibana 4 dashboards but allow anyone to view dashboards without authentication. I recently installed an ELK (Elasticsearch 1.4.5, ...
3
votes
1answer
4k views

Large virtual memory size of ElasticSearch JVM

I am running a JVM to support ElasticSearch. I am still working on sizing and tuning, so I left the JVM's max heap size at ElasticSearch's default of 1GB. After putting data in the database, I find ...
0
votes
1answer
303 views

CentOS7 + rsyslog-elasticsearch.x86_64 + nginx = charset=utf-8] is not supported

I am building a tiny system for log analysis, and not able to install additional agent, so using already working rsyslog to send logs to elasticsearch. My rsyslog and elasticsearch are separate VM's. ...
3
votes
1answer
854 views

Why is this exclude_lines in filebeat excluding all logs?

I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-...
0
votes
0answers
284 views

centos dedicated server backup strategy for mongodb and elasticsearch db

I have a main server (dedicated CentOS 7, XFS file system, 2X1TB SSD, software RAID1) connected via LAN to my backup server (dedicated CentOS 7, XFS file system, 2X2TB HDD, software RAID1) in my setup....