Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

Filter by
Sorted by
Tagged with
2
votes
1answer
55 views
+50

Elasticsearch: how to “rescue” the documents that could not be parsed by mapping?

We're using ElasticSearch to store and inspect logs from our infrastructure. Some of those logs are required by law, and we can't afford to lose any. We've been parsing logs for quite some time ...
0
votes
0answers
39 views

elastic search behind haproxy

There are 4 elastic search nodes in a cluster and these 4 nodes are setup behind haproxy. Each of the node in the cluster is receiving around 900 to 1000 tcp connections, but one of the node has very ...
1
vote
1answer
1k views

Signing/Generating SSL Certificates for ElasticSearch (search-guard)

I am trying to build my own ElasticSearch cluster and I decided to use an open source plugin called Search Guard. It requires generation of client and host/node certificates. In its demo there is a ...
0
votes
0answers
11 views

APM PHP Agent Setup

Im having a really hard time setting up my APM PHP Agent on a test EC2 server, can anyone please show me how to configure it properly to send log data to my ELK Server. I tried asking this question on ...
3
votes
3answers
3k views

kibana: no indices error

None of the existing answers helped, so here a new question. Usecase: Redirecting syslog (or) monitoring static file. I have successfully installed logstash (1.4.2), elasticsearch(1.1.1) and kibana(...
1
vote
1answer
1k views

memory locking requested for elasticsearch process but memory is not locked

ElasticSearch 5.6.7 throws somewhat misleading following error and refuse to start: [2018-08-02T10:07:50,333][ERROR][o.e.b.Bootstrap ] [esm56n3] node validation exception [2] bootstrap ...
3
votes
2answers
2k views

How to increase disk size in a stateful set

I'm managing an Elasticsearch deployment in Kubernetes. I see that the disk storage is getting close to full, so I would like to increase the size of the persistent volumes. I want to change this ...
0
votes
1answer
1k views

Why doesn't logstash grab or index the files from the mapped drive?

I don't understand why logstash is so finicky with network resources. I shared a folder on another machine and then mapped it as Z: under Windows Explorer. I've verified the path and everything. I can ...
0
votes
0answers
49 views

How to find URL for azure active directory to integrate with elasticsearch xpack?

I want to integrate my default active directory in azure with my elasticsearch server. For integration azure active directory domain URL is required. I tried by specifying the domain in elasticsearch ...
1
vote
0answers
21 views

init: logstash main process (19281) terminated with status 1

I run Amazon Linux instance with logstash 6.4.0. [root@****** dragan]# cat /etc/issue Amazon Linux AMI release 2017.03 I wanted to upgrade logstash so I uninstalled 6.4.0 and tried to install 7.3.2. ...
6
votes
2answers
5k views

Is there a way to stop the AWS Elasticsearch?

I setup an AWS Elasticsearch Domain recently but I didn't see a way to stop it (like you can with an EC2 instance), which means I'm continuously billed. At this stage I just need to do some testing ...
1
vote
0answers
18 views

Is it safe to allow POST access to the multi search API of ElasticSearch

We are running an application using elasticsearch hosted on AWS. The client-side interface uses "ReactiveSearch" for vuejs, which makes POST requests against the "_msearch" interface described here: ...
1
vote
2answers
782 views

How to move/migrate indices from self hosted Elasticsearch to AWS Elasticsearch service (cloud)

I have a self hosted Elasticsearch 6.2 cluster (2 master nodes, ~200Gb data each). I plan to move to AWS Elasticsearch service & it's not possible to ssh into it. What's the fastest way to move ...
0
votes
0answers
31 views

ELK Cluster Minimum Node Count

Can we build ELK Cluster with one master and one data node, 2 nodes cluster in total? I need replication of shards. Just in case one of the node is down, does master also keep a copy of shards or is ...
0
votes
1answer
2k views

Kibana @timestamp mapping & filter

I'm using following system/package: $ cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) $ rpm -q filebeat filebeat-1.3.0-1.x86_64 $ with /etc/filebeat/filebeat.yml: $ cat /etc/...
0
votes
1answer
3k views

elasticsearch: max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536]

When I tried to run the logging aggregation I found out the following error generated by elasticsearch: [2018-02-04T13:44:04,259][INFO ][o.e.b.BootstrapChecks ] [elasticsearch-logging-0] bound or ...
0
votes
0answers
16 views

logstash not proccessing filebeat data

I have configured elk stack on a aws server and also configured filebeat on two different aws server with same config. Telnet from both filebeat server to elk server on port 5044 working fine but ...
0
votes
1answer
1k views

Error while setting up grafana with graphite and nginx

While trying to setup grafana with graphite using nginx server. I am getting this error: No 'Access-Control-Allow-Origin' header is present on the requested resource. I tried changing configuration ...
0
votes
1answer
29 views

Elassandra “action_request_validation_exception”

Elasticsearch DEBUG: 2019-09-23T09:23:39Z starting request { "method": "HEAD", "castExists": true, "path": "/my_keyspace", "query": {} } Elasticsearch TRACE: 2019-09-23T09:23:39Z ...
1
vote
0answers
25 views

Elasticsearch is pointing to only one node on the server

I am using Elasticsearch 6.4.0 and it's working fine, except that there are 3 nodes on the server but my service points only to one node. This causes a slowdown in performance. How can I point my ...
2
votes
0answers
130 views

Logstash to aggregate logs into ElasticSearch

I am trying to aggregate linux logs using rsyslog into Logstash/ElasticSearch running in EKS. filebeat is already running in EKS to aggregate Kubernetes container logs. I have configured rsyslog ...
0
votes
1answer
72 views

Unable to access ElasticSearch on AWS EC2 with private/public ip

I have an Elastic Search running on 9200 on rhel 6. I cannot reach to it in aws with private/public ip. On the other hand, i try it with curl localhost:9200, it is good. I can also ping google. I ...
2
votes
2answers
991 views

How to install Elasticsearch plugins with docker container?

I am a newbie in Docker, I want to install plugins in my container of elasticsearch, in this case they are: analysis-icu analysis-phonetic I know that in a traditional way it would be like this: /...
1
vote
1answer
2k views

cannot validate certificate - doesn't contain any IP SAN

I am currently in the process of installing ELK ( ElastricSearch, LogStash & Kibana) stack. My ELK server IP address is 172.29.225.32. Elastic Search config is :: # -----------------------------...
0
votes
0answers
20 views

AWS, Nginx : Redirect data from a ElasticSearch domain to nginx domain

We have our app-code running on AWS, inside a non-default VPC. All the logs are redirected to an ElasticSearch instance. There is a Kibana URL available, but it's only accessible if you do local port-...
0
votes
2answers
3k views

What are records in 'netstat' command are when using logstash?

I use logstash with embedded elasticsearch. My logstash config looks like the following: output { tdout { debug => true debug_format => "json"} elasticsearch { embedded => true ...
0
votes
1answer
140 views

Error from server (Timeout): error when creating “STDIN”: Timeout: request did not complete within requested timeout 30s

I'm trying to Deploy the Elasticsearch cluster, yet running into following error (timeout): $ cat <<EOF | kubectl apply -f - > apiVersion: elasticsearch.k8s.elastic.co/v1alpha1 > kind: ...
0
votes
0answers
152 views

How to enable elasticsearch 6.8 deprecation logging? And where can I find it?

Ultimately, I want to upgrade a bunch of Python code that uses Elasticsearch 6.8 to instead use Elasticsearch 7.3. Toward that goal, I want to enable Elasticsearch 6.8 "deprecation logging". It ...
0
votes
1answer
1k views

How to provision dashboards into kibana?

I am giving my developers the gift of kibana using the elasticsearch that is bundled with logstash. They will have kibana in their vagrant boxes. Goal: Install and configure kibana dashboards via ...
5
votes
2answers
4k views

“Index Patterns: Please specify a default index pattern” in Kibana

I'm trying to create a simple hello world for ELK and be able to see kibana reports via the internet. I've installed kibana, logstash, nginx and elastic search. Here's my /etc/logstash/conf.d/10-...
0
votes
1answer
4k views

Elastic Search Unassigned Shards

my cluster state was green and after restart of service one shard remains UNASSIGNED and status goes to yellow. I have 2 machines with 5 shard and 1 replica set settings. I am using default config ...
4
votes
2answers
2k views

Elasticsearch performance tuning

In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge. Logstash conf ...
1
vote
1answer
1k views

iptables to allow input and output traffic to and from web server only

I have an Elastic Search server which seems to have been exploited (it's being used for a DDoS attack having had NO firewall for about a month). As a temporary measure while I create a new one I was ...
0
votes
3answers
40 views

t2.micro elasticsearch7.3 not working after editing config file

elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendo Active: failed (Result: exit-code) since Tue 2019-08-27 02:13:03 UTC; 3min 47 ...
0
votes
0answers
15 views

rsyslog: “No local error logger defined”

I'm using rsyslog to parse logs from haproxy and send them to elasticsearch. Some logs will fail to be sent to elasticsearch for various reasons. We want to keep those logs somewhere. As suggested by ...
0
votes
1answer
54 views

rsyslog: Specify `action` parameters from environment variables

I'm using rsyslog to send logs to elasticsearch. It all works well in my local environment, but now I'm trying to make it more generic and inject environment variables where needed. As part of my ...
0
votes
0answers
14 views

Filebeat custom log files / how to disable default module field mappings

I'm using filebeat to read log files that are not supported out of the box, for elasticsearch indexing. The thing is that I get 1000+ field mappings that appear to be coming from default filebeat ...
1
vote
1answer
2k views

vm.max_map_count problems on GKE ElasticSearch StatefulSet

A problem appeared on working ElasticSearch cluster on GKE. Nodes with "data" roles began to crash unexpectedly with an error: max virtual memory areas vm.max_map_count [65530] is too low, increase ...
1
vote
1answer
3k views

Occasional “failed to respond” from AWS hosted Elasticsearch

We have an Elasticsearch cluster hosted on Amazon Elasticsearch Service (AWS). We are using the Jest Java HTTP Rest client for ElasticSearch. Every now and then (perhaps 1 in 10,000 requests), it ...
1
vote
1answer
26 views

Documents from Elasticsearch Cluster appearing twice in Grafana

I set up an Elasticsearch cluster with one dedicated master node, two master-eligible data nodes and one coordinating node. The number of replicas is set to one. There are two pipelines in Logstash, ...
0
votes
3answers
428 views

ElasticSearch doesn't start after installing 5.6.4

I just installed ElasticSearch 5.6.4 from https://www.elastic.co/downloads/elasticsearch via DEB (I use Debian 9) package using these commands: wget https://artifacts.elastic.co/downloads/...
0
votes
1answer
26 views

regular timeout error at certain time period

My system sends the data to Redis via web server and ultimately stacks it in Elasticsearch. The thing is that Redis connection timeout error occurs when the traffic is small. The error always occurs ...
0
votes
0answers
28 views

How do i send all logs from docker swarm to Elastic?

I have number of Linux server that have docker installed on them, all of the server are in a docker swarm, on each server i have a custom application. I also have ELK setup in AWS. I want to collect ...
0
votes
0answers
121 views

Provided Grok expressions do not match field value

edit: read below for update to initial question I'm getting Provided Grok expressions do not match field value even though _simulate works with exact same string. Error message: "message": "172.18....
0
votes
1answer
370 views

Elasticsearch connection refused

I have just installed ElasticSearch 7.1.1 on Debian 9 throw apt-get repository VPS 4GB ram .. 1vcpu service elasticsearch status elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/...
0
votes
1answer
190 views

How to fix unable to connect to ldapserver - Authentication Finally Failed

I'm using openldap on opendistro for elasticsearch with docker I get this error: elasticsearch | [2019-07-31T12:48:42,590][WARN ][c.a.o.s.a.BackendRegistry] [28da1860f0c0] Authentication ...
0
votes
0answers
100 views

logstash not able to upload data to elasticsearch even the pipeline started

I am using elasticsearch 7.1.1 and logstash 7.1.1. I am trying to upload a log file to elastic search using grok filter. Pipeline is getting started, but data is not getting uploaded. Here is my ...
0
votes
0answers
91 views

Bitbucket fails to start on Ubuntu 16.04 LTS

I installed Bitbucket 5.3.1 from the terminal, but when I attempt to startup the app it hangs with the following log output: INFO [spring-startup] c.a.s.i.hazelcast.HazelcastLifecycle Updating ...
0
votes
1answer
35 views

logstash and rsyslog sends wrong hostname

I'm setting up a centralised logging system, comprising two VM's. Logger, which acts as the central syslog collector (using rsyslog) and Logstash processor; and Searcher, which acts as the ...
0
votes
1answer
60 views

Elasticsearch 404 attempting to create index through nginx proxy

I am attempting to proxy Elasticsearch through nginx. Most things seem to work, but one thing that is having problems is creating indexes. This works when connecting directly to ES: curl -XPUT http:...