Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

Filter by
Sorted by
Tagged with
0
votes
2answers
28 views

Kibana and Logstash: When is a field not a field?

I've enabled the grok filter in our logstash configuration in order to parse logs from HAProxy using the %{HAPROXYHTTP} and %{HAPROXYTCP} patterns. This seems to work great and viewing the details for ...
0
votes
0answers
9 views

Scaling Elasticsearch down to single-node

Is it possible to scale Elasticsearch from multiple nodes down to one node? I have a 3-node cluster that is way overkill for the amount of data being logged. To scale it down, I set "cluster.routing....
0
votes
0answers
33 views

Does Elasticsearch have capability to form ES cluster with only one ES docker service running multiple replicas of it in Docker-Swarm

I wanna run only one elasticsearch service in docker-swarm having multiple replicas of this service. Does Elasticsearch have capability to form ES cluster with only one ES docker service running ...
0
votes
0answers
9 views

Auto_attribute plugin like discovery_ec2 that works on GCE?

The discovery_ec2 has a very nice feature that populates the attributes for a node based on the EC2 metadata: https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-ec2-usage.html#...
0
votes
0answers
18 views

Elasticsearch snapshot can not access directory

I am trying to get elasticsearch to make regular snapshots on Red Hat 8. I started by adding path.repo: "/home/user/backup" to the configuration file. As a test. When I restart the service, it ...
0
votes
0answers
22 views

Index rollover latency

I have created an ILM policy as follows, { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_docs": 30 }, ...
0
votes
1answer
22 views

Do we need a back up of Elastic Search data in a database in case of failure, or is there a solid permanant storage solution with Elastic Search?

We are looking to use Elastic Search in a major software system, which needs financial logs and records kept for many years. We are using Elastic Search, but are a little concerned whether Elastic is ...
1
vote
1answer
52 views

Virtual machine disk I/O latency spikes

Since past few days I have strange I/O spikes in one virtual machine. Its 2.6.32-504.el6.x86_64 #1 SMP Tue Sep 16 01:56:35 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server ...
0
votes
0answers
18 views

Restricting Elasticsearch with Nginx Basic auth

I have an Elasticsearch instance running on a box at port 9200. I am trying to secure it with Nginx which I am new to. I have running Elasticsearch running on port 9200 and here is my nginx config to ...
0
votes
1answer
20 views

Elasticsearch - how to disable client access on data node?

I am fairly new to Elasticsearch and was watching some tutorials that are somewhat outdated. In the tutorial, they said that if one has dedicated data nodes, it is recommended to disable client ...
0
votes
0answers
31 views

Getting log rotation to work with Elasticsearch and Log4j2

We recently discovered that our ES 6.5 servers never purge old log files. Based on various information found on google I came up with this configuration for log4j2, which we applied to the servers ...
0
votes
0answers
20 views

Auto login for kibana visualizations not working, nginx

Hi im trying to use nginx to auto authenticate in kibana shared visualizations and it isnt working. Im on a centos 8, installed nginx, installed httpd-tools ,I create a folder called: sites-available,...
0
votes
1answer
106 views

Integrating nginx logs and elasticsearch app-search

I'm trying to setup a self-managed docker appsearch instance, together with kibana and elasticsearch, queried by a uvicorn python app, proxied by a nginx webserver My current issue is that the ...
0
votes
1answer
98 views

Pi4+ runs Elasticsearch cluster of 1, gives yellow status

I am currently running Elasticsearch 7.3.2 on a Rasbian (Buster) instance on a Raspberry Pi 4+. It looks like it was running in a green status for a few days, processing files, but suddenly, i ...
0
votes
0answers
15 views

ElasticSearch : LDAP authentication on local node and processing on cloud node

Our LDAP server is restricted to requests from the intranet. But we would like to run ElasticSearch with LDAP on cloud. Is it a good strategy to run a local node of ElasticSearch locally within the ...
0
votes
1answer
91 views

AWS ElasticSearch service does not provide Kibana users management

First I'm going to explain my current architecture. I have an EC2 instance running with a Logstash daemon, as input an http configuration and as output my AWS ElasticSearch instance. As I said before,...
0
votes
0answers
111 views

fluentd correct mapping and indexing of @timestamp

The timestamp in Kibana is a long field (containing a unix timestamp) with the following configuration: <source> @type tail path /logs/other_vhosts_access.log pos_file /logs/access.log....
0
votes
0answers
122 views

ELK - Logstash not picking up syslog events

I'm setting up a ELK cluster using Centos 8 and version 7.4 of Elasticsearch, Logstash and Kibana. My issue is with Logstash not picking up the events coming through syslog. Configuring Logstash to ...
0
votes
0answers
188 views

Run OCR in Nextcloud with Docker

I'm using the official Docker image nextcloud:latest to run my Nextcloud instance. I managed to install Elasticsearch (on the host) and created an index. Search works so far. Now I want to OCR PDFs ...
0
votes
0answers
74 views

elastic search behind haproxy

There are 4 elastic search nodes in a cluster and these 4 nodes are setup behind haproxy. Each of the node in the cluster is receiving around 900 to 1000 tcp connections, but one of the node has very ...
0
votes
0answers
39 views

APM PHP Agent Setup

Im having a really hard time setting up my APM PHP Agent on a test EC2 server, can anyone please show me how to configure it properly to send log data to my ELK Server. I tried asking this question on ...
0
votes
0answers
63 views

How to find URL for azure active directory to integrate with elasticsearch xpack?

I want to integrate my default active directory in azure with my elasticsearch server. For integration azure active directory domain URL is required. I tried by specifying the domain in elasticsearch ...
1
vote
0answers
45 views

init: logstash main process (19281) terminated with status 1

I run Amazon Linux instance with logstash 6.4.0. [root@****** dragan]# cat /etc/issue Amazon Linux AMI release 2017.03 I wanted to upgrade logstash so I uninstalled 6.4.0 and tried to install 7.3.2. ...
1
vote
0answers
30 views

Is it safe to allow POST access to the multi search API of ElasticSearch

We are running an application using elasticsearch hosted on AWS. The client-side interface uses "ReactiveSearch" for vuejs, which makes POST requests against the "_msearch" interface described here: ...
0
votes
0answers
36 views

ELK Cluster Minimum Node Count

Can we build ELK Cluster with one master and one data node, 2 nodes cluster in total? I need replication of shards. Just in case one of the node is down, does master also keep a copy of shards or is ...
0
votes
0answers
17 views

logstash not proccessing filebeat data

I have configured elk stack on a aws server and also configured filebeat on two different aws server with same config. Telnet from both filebeat server to elk server on port 5044 working fine but ...
0
votes
1answer
61 views

Elassandra “action_request_validation_exception”

Elasticsearch DEBUG: 2019-09-23T09:23:39Z starting request { "method": "HEAD", "castExists": true, "path": "/my_keyspace", "query": {} } Elasticsearch TRACE: 2019-09-23T09:23:39Z ...
1
vote
0answers
26 views

Elasticsearch is pointing to only one node on the server

I am using Elasticsearch 6.4.0 and it's working fine, except that there are 3 nodes on the server but my service points only to one node. This causes a slowdown in performance. How can I point my ...
0
votes
1answer
362 views

Unable to access ElasticSearch on AWS EC2 with private/public ip

I have an Elastic Search running on 9200 on rhel 6. I cannot reach to it in aws with private/public ip. On the other hand, i try it with curl localhost:9200, it is good. I can also ping google. I ...
2
votes
0answers
158 views

Logstash to aggregate logs into ElasticSearch

I am trying to aggregate linux logs using rsyslog into Logstash/ElasticSearch running in EKS. filebeat is already running in EKS to aggregate Kubernetes container logs. I have configured rsyslog ...
0
votes
0answers
38 views

AWS, Nginx : Redirect data from a ElasticSearch domain to nginx domain

We have our app-code running on AWS, inside a non-default VPC. All the logs are redirected to an ElasticSearch instance. There is a Kibana URL available, but it's only accessible if you do local port-...
0
votes
0answers
230 views

How to enable elasticsearch 6.8 deprecation logging? And where can I find it?

Ultimately, I want to upgrade a bunch of Python code that uses Elasticsearch 6.8 to instead use Elasticsearch 7.3. Toward that goal, I want to enable Elasticsearch 6.8 "deprecation logging". It ...
0
votes
3answers
47 views

t2.micro elasticsearch7.3 not working after editing config file

elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendo Active: failed (Result: exit-code) since Tue 2019-08-27 02:13:03 UTC; 3min 47 ...
0
votes
0answers
16 views

rsyslog: “No local error logger defined”

I'm using rsyslog to parse logs from haproxy and send them to elasticsearch. Some logs will fail to be sent to elasticsearch for various reasons. We want to keep those logs somewhere. As suggested by ...
0
votes
0answers
33 views

Filebeat custom log files / how to disable default module field mappings

I'm using filebeat to read log files that are not supported out of the box, for elasticsearch indexing. The thing is that I get 1000+ field mappings that appear to be coming from default filebeat ...
0
votes
1answer
118 views

rsyslog: Specify `action` parameters from environment variables

I'm using rsyslog to send logs to elasticsearch. It all works well in my local environment, but now I'm trying to make it more generic and inject environment variables where needed. As part of my ...
1
vote
1answer
29 views

Documents from Elasticsearch Cluster appearing twice in Grafana

I set up an Elasticsearch cluster with one dedicated master node, two master-eligible data nodes and one coordinating node. The number of replicas is set to one. There are two pipelines in Logstash, ...
0
votes
1answer
26 views

regular timeout error at certain time period

My system sends the data to Redis via web server and ultimately stacks it in Elasticsearch. The thing is that Redis connection timeout error occurs when the traffic is small. The error always occurs ...
0
votes
0answers
61 views

How do i send all logs from docker swarm to Elastic?

I have number of Linux server that have docker installed on them, all of the server are in a docker swarm, on each server i have a custom application. I also have ELK setup in AWS. I want to collect ...
0
votes
0answers
305 views

Provided Grok expressions do not match field value

edit: read below for update to initial question I'm getting Provided Grok expressions do not match field value even though _simulate works with exact same string. Error message: "message": "172.18....
0
votes
1answer
200 views

Error from server (Timeout): error when creating “STDIN”: Timeout: request did not complete within requested timeout 30s

I'm trying to Deploy the Elasticsearch cluster, yet running into following error (timeout): $ cat <<EOF | kubectl apply -f - > apiVersion: elasticsearch.k8s.elastic.co/v1alpha1 > kind: ...
2
votes
2answers
174 views

Elasticsearch: how to “rescue” the documents that could not be parsed by mapping?

We're using ElasticSearch to store and inspect logs from our infrastructure. Some of those logs are required by law, and we can't afford to lose any. We've been parsing logs for quite some time ...
0
votes
1answer
428 views

How to fix unable to connect to ldapserver - Authentication Finally Failed

I'm using openldap on opendistro for elasticsearch with docker I get this error: elasticsearch | [2019-07-31T12:48:42,590][WARN ][c.a.o.s.a.BackendRegistry] [28da1860f0c0] Authentication ...
0
votes
1answer
83 views

logstash and rsyslog sends wrong hostname

I'm setting up a centralised logging system, comprising two VM's. Logger, which acts as the central syslog collector (using rsyslog) and Logstash processor; and Searcher, which acts as the ...
0
votes
1answer
77 views

Elasticsearch 404 attempting to create index through nginx proxy

I am attempting to proxy Elasticsearch through nginx. Most things seem to work, but one thing that is having problems is creating indexes. This works when connecting directly to ES: curl -XPUT http:...
3
votes
1answer
887 views

Bootstrapping a 3 node Elastic Search cluster using docker containers

I'm trying to set up a test Elastic Search cluster on 3 separate hosts, using the official 7.2.0 docker image Each container is is configured with an elasticsearch.yml which looks like this cluster....
3
votes
2answers
3k views

How to install Elasticsearch plugins with docker container?

I am a newbie in Docker, I want to install plugins in my container of elasticsearch, in this case they are: analysis-icu analysis-phonetic I know that in a traditional way it would be like this: /...
0
votes
0answers
145 views

Creating email alerts for kibana filters via sentil

I have a elk stack that works properly and can be monitored via the kibana dashboard. But as an improvement to the whole setup I have being researching for a reporting tool to setup with kibana. My ...
0
votes
0answers
30 views

Determine running version of elasticsearch, without using http interface

If an elasticsearch node supports the http interface, you can check the running version via curl -XGET 'localhost:9200' (or whatever port the http interface is on). If you want to know the installed ...
1
vote
0answers
265 views

What is causing the Elasticsearch service to ignore my jvm.options?

In my /etc/elasticsearch/jvm.options I have set my heap size with: -Xms256m -Xmx256m However when I restart the service, memory keeps rising until it runs out and the service is stopped abruptly: [...

1
2 3 4 5
7