Questions tagged [elasticsearch]
Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.
379
questions
0
votes
0
answers
42
views
Connecting multiple PHP containers to a single shared Elasticsearch container
14 PHP containers php_1, ..., php_14 and 14 NGINX containers nginx_1, ..., nginx_14 are running on a server. The PHP container php_i is only connected to nginx_i and all NGINX containers are connected ...
0
votes
1
answer
47
views
How to set kubernetes secrets outside a container section
I have a generated kubernetes secret like this as it is shown e.g. in argocd:
apiVersion: v1
data:
ES_GUEST_PASSWORD: ++++++++
ES_GUEST_USERNAME: ++++++++
kind: Secret
I now want to use it as an ...
0
votes
1
answer
45
views
How to gather application logs which are generating in custom path using EFK
I created the elasticsearch and kibina setup which is running outside the kubernetes cluster and i am using fluentd to gather kubernetes logs and to send data to elasticsearch. I am running fluentd as ...
0
votes
1
answer
547
views
Fluent-bit in Kubernetes cannot input timestamp data properly
At my company, I built a K8s cluster with Terraform and configured a logging system with EFK (Elasticsearch, Fluent-bit, Kibana).
k8s and Elasticsearch use AWS's EKS and Opensearch Servcie (ES 7.10), ...
0
votes
1
answer
600
views
Could not contact DNS servers (fluent bit)
I am configuring the fluent bit connection with elasticsearch, I have 3 elasticsearch nodes and I want to have high availability, according to the following documentation it is done like this text, ...
0
votes
0
answers
98
views
Fluentd logs not sent to Elasticsearch on K8s
I have a cluster in VirtualBox to learn kubernetes. I have a deployment that contains MySQL and phpMyAdmin. I created a DemonSet that has the fluentd image and collects the logs to transmit them to ...
0
votes
0
answers
281
views
Fluentd logs not sent to Elasticsearch
I have a cluster in VirtualBox to learn kubernetes. I have a deployment that contains MySQL and phpMyAdmin. I created a DemonSet that has the fluentd image and collects the logs to transmit them to ...
0
votes
0
answers
77
views
How do I shutdown a specific node in ElasticSearch?
I'm running ElasticSearch 7.3.2 and I found this page documenting how to shut down a node: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/put-shutdown.html
I tried running this command (...
0
votes
0
answers
364
views
High CPU usage of kube-controller and kube-apiserver when increasing nodes of ECK node sets
I am trying to run an Elastic Cluster on Kubernetes (Minikube). It basically works, but only with one server per node. As soon as I try to set up > 1 node per node set, the cpu usage of
kube-...
0
votes
0
answers
623
views
Running Opensearch behind a proxy
when I configure nginx to redirect /monitor to opensearch, opensearch receives the requests and redirects the user to /app/something which is not under /monitor and thus fails (error 404).
My nginx ...
0
votes
0
answers
435
views
Could not communicate to Elasticsearch, resetting connection and trying again. EOFError (EOFError)[Fluentd using helm on kubernetes]
I'm installing elasticsearch, kibana and fluentd in kubernetes with helm chart Elasticsearch and kibana pods stand up smoothly, but fluentd pods don't stand up I get the following errors:
2023-06-13 ...
0
votes
0
answers
293
views
How to have multiple instances of filebeat load balance Netflow input?
I have a very high volume Netflow input stream, and I was hoping that I could run multiple instances of Filebeat and load-balance the Netflow traffic over the Filebeat instances, and then write to a ...
0
votes
0
answers
305
views
Filebeat docker multiline
I'm using filebeat to retrieve log files to Elastic cloud. I'd like to put errors and java exception in one document instead of multiple for each line. I managed to make it work on text log files but ...
0
votes
1
answer
582
views
Can I use keycloak as an Idp for kibana installed using the ElasticSearch Operator in kubernetes?
I have setup a kubernetes cluster in a private network and I'm using a gateway machine for accessing the cluster. In the kubernetes cluster I have installed the elasticsearch operator and through that ...
0
votes
1
answer
9k
views
Elasticsearch: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors (ca.crt matches on all nodes)
Note that this entire cluster was working fine up 'til I updated certificates. I'm on Elasticsearch 7.5.
I updated my certificates using the certificate-util - created a server.yml with all my servers ...
2
votes
0
answers
641
views
rsyslog is not forwarding logs to elasticsearch
I'm trying to configure rsyslog to send logs to logstash and then forward them to elasticsearch.
I have create a config file /etc/rsyslog.d/60-output.conf with the following content:
*.* @localhost:...
0
votes
1
answer
675
views
SonarQube is not starting on SynologyNAS due to CONFIG_SECCOMP
SonarQube is unable to start on Synology NAS Docker due to the CONFIG_SECCOMP not being present on Synology NAS. Is there an option to ignore SECCOMP for SonarQube while it's starting ElasticSearch?
...
0
votes
1
answer
319
views
Restricting Kibana to TLSv1.2
What is the setting that needs to be set in the kibana.yml file (or other file) to restrict Kibana to TLSv1.2 over HTTP?
I believe that it uses TLSv1.0, TLSv1.1, and TLSv1.2 by default. I need to ...
1
vote
0
answers
71
views
Suricata / Filebeat / ELK - iptables tee - Create virtual hosts
I have an IDS setup as follow:
Hardware / interfaces
WAN <----(brwan)> ROUTER / AP <(br0)----> LAN
\
-----(eth1)>...
1
vote
1
answer
101
views
Handle 150 req/s and 2TB of logs for 7 days of retention available by API
I would like to get advise from experienced people to build a HA infrastructure to log 2To of data in JSON format every week.
I need to have a retention time of 7 days and need to be able to requests ...
0
votes
1
answer
3k
views
curl -X GET http://localhost:9200 return Access Denied
I am trying to install and setup Elasticsearch 8 on Centos 7 server, the service started but when I do curl -x GET HTTP://localhost:9200, or curl -x GET HTTP://127.0.0.1:9200 or curl -x GET HTTP://(...
0
votes
0
answers
1k
views
Free storage space in amazon open search data node is low and others nodes has more than enough free space
In my AWS open search i am using 8 data nodes (r5.2xlarge.search) and 3 master nodes (r5.large.search) and each data nodes has allocated to a space of 200gb each. And also added my shard value as 4....
0
votes
1
answer
482
views
Graylog - data from newest indice not returned on search
I have a Graylog server (newest version) collecting data from nginx. It´s been running for a while. I retrieve the collected data by querying my Elasticsearch cluster (v7 newest version) which ...
1
vote
0
answers
148
views
Amazon OpenSearch Service stably or less downtime?
Can anyone point out, What is the best way for the elastic search to run stably? We frequently get messages about it requiring updates and that an update requires some downtime.
Could we have 2 ...
1
vote
0
answers
376
views
On Debian system, keepalived is consuming full cpu usage on both kibana nodes
I have master/slave kibana machines on Debian 10. From few days, cpu usage of keepalived daemon on both nodes are reaching to 100%. After keepalived restart it goes to normal, and this loops forever.
...
0
votes
0
answers
38
views
How to fetch Individual output of different match_phrase inside one request
I am able to run below query which is giving response as total of two match_phrase.
Inserting dummy data as below.
POST /mod1/_bulk
{ "index" : { } }
{ "msg": "BA1" }
{ &...
0
votes
1
answer
744
views
How to pass values to query externally
I have below similar logs.
I have created dummy index and created mapping like below in dev-tools
PUT new
{
"mappings": {
"properties": {
"@timestamp": {
...
0
votes
1
answer
138
views
Can logstash "pull" data?
I have two servers. Server A is running Elasticsearch and Logstash. Server B is running filebeat and is also the server which contains all the logs I'm trying to analyse.
Server A is behind a firewall,...
0
votes
0
answers
612
views
Finding source of high IOWait
I have a 5 node elasticsearch cluster. One host has had consistently high IOWait% (40+) for several weeks. The others seem fine (<10%).
Notable:
Host in question is not the cluster master
New ...
1
vote
1
answer
405
views
graylog 4.0 - Injector could not be created
Trying to set up graylog with official graylog2.graylog ansible role.
Installed Elasticsearch (v7.10.2) cluster on three separate servers with official elasticsearch ansible role (node_1 - master, ...
1
vote
0
answers
1k
views
My elastic search is not showing any data using elastic-agent
Surprisingly my elastic version 7.15 with basic license have installed the fleet server and then configured windows-policy with elastic-endpoint and windows integration. Have enabled all the rules; ...
1
vote
1
answer
615
views
Running Enterprise Search as a service
Elastic's Enterprise Search Installation Guide goes as far as starting the process locally - which is obviously not very stable.
When installed from a .deb or .rpm, a service is actually set up and ...
1
vote
1
answer
620
views
Is there a way to use the ingest attachment plugin with Elastic App Search
I'm working on a portal that hosts multiple types of documentation (HTML, PDF, PPTx, DocX) and makes them all searchable in one place.
We could achieve this using the "standard" out of the ...
1
vote
1
answer
836
views
Kibana does not trust Let's Encrypt CA in Kubernetes Helm Setup
I have a k8s cluster configured with calico as networking system. I'm running slightly customized versions of helm elastic/elasticsearch and elastic/kibana with security enabled. For security, I use ...
2
votes
1
answer
3k
views
When to use a new index in Graylog (Elasticsearch)?
I have been searching for days now to find a good explanation on how indices are used by Graylog/Elasticsearch and when to create a new one. There is a lot of info about sharding indices but not much ...
0
votes
1
answer
273
views
Filebeat on ECK with AWS Module Fails Due To Metadata Error
We are running an Elastic Stack with ECK in EKS (7.8). We noticed that our filebeat daemonset and the AWS module were not processing logs from S3 and our SQS queues backing up. Looking at the logs on ...
0
votes
1
answer
9k
views
Elasticsearch - Can I delete a data-stream backing index?
I have a few data-streams configured in my Elasticsearch cluster (with ILM policy).
Suppose I run into storage problems and etc. and I want to prematurely delete a few backing indices of the data-...
0
votes
1
answer
11k
views
Kibana cannot be accessed through port 5601 in browser [duplicate]
I cannot get to kibana through web UI. I'm set the necessary ports in firewall-cmd, I've configured kibana.yml and elasticsearch.yml. I'm not seeing any relevant errors, or at least don't know if they ...
0
votes
0
answers
71
views
Multiple Web Application - Cache Layer Design
I have multiple client e-commerce web applications running on a single VM, each web application is a node.js express application.
The express web application communicates with the back end via API's ...
0
votes
1
answer
560
views
How to stop ElasticSearch background task
According to the CLI help of ElasticSearch, you can start ElasticSearch in the background with elasticsearch -d or elasticsearch --daemonize. This is what I use all the time locally, as it seems to ...
0
votes
0
answers
35
views
Ubuntu server 'partially' responsive
I have an Ubuntu 18.04 server that has periodically unresponsive - I can ping it, and it appears that some https services are running (albeit v. slowly), however it hangs if I try to log in (on the ...
0
votes
1
answer
66
views
realtime monitoring of linux server network
We have a few Linux server which act as Edge network, We want realtime (or near-realtime) information on network bandwidth, top IP addresses and network bandwidth per them and other network parameters ...
0
votes
1
answer
129
views
How to display the logs of a VM on ELK stack which is running on another VM?
-ELK is running on localhost & successfully getting logs.
-Both VM's are on same server.
-OS on both VM'S is Ubuntu 18-04 LTS.
0
votes
1
answer
587
views
NGINX - Access port via SSL
I have ElasticSearch and NGINX set up on my server. Accessing it via SSL is failing.
I can access ElasticSearch fine via:
http://ipaddress:9200
http://mydomain.co.uk:9200
The domain was then set up ...
0
votes
0
answers
38
views
Storage balancing in elasticsearch cluster
I've a Elasticsearch cluster with 5 nodes. I've this used storage repartition:
Node1: 76%
Node2: 94%
Node3: 88%
Node4: 73%
Node5: 74%
How to do to storage balancing/leveling the used storage ?
For ...
1
vote
0
answers
342
views
Correct reverse proxy to aws-es-proxy instance?
I installed a VPC with an Elasticsearch cluster with kibana on AWS.
As that is VPC, it is not accessible publicly.
Thus, to be able to access the kibana interface, I installed the https://github.com/...
0
votes
1
answer
184
views
How to use filebeat to find password in log
I have an Elasticsearch 7.1 and i have configure filebeat to collect all log.
I want to check if i have password in log.
So anyone have an idea how can i find all password in log using filebeat.
Thank ...
1
vote
2
answers
4k
views
Can you help me understand this ElasticSearch permissions error?
I'm attempting to make a request to an Elastic.co hosted ElasticSearch instance. I'm receiving this error:
AuthorizationException(403, 'security_exception', 'action [indices:monitor/stats] is ...
0
votes
1
answer
964
views
Portainer: Pass vm.max_map_count to SonarQube Docker container
I need to set up a SonarQube container on my Portainer instance. SonarQube uses Elasticsearch, which needs a kernel setting passed from the host. This issue is well documented, as mentioned in this ...
0
votes
1
answer
12k
views
Kibana - "missing authentication credentials for REST request"
I have a elasticsearch cluster with xpack basic license, and native user authentication enabled (with ssl of course). I am attempting to set up kibana on a docker container but keep getting an error ...