Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
The results are in! See what nearly 90,000 developers picked as their most loved, dreaded, and desired coding languages and more in the 2019 Developer Survey.

Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

4
votes
0answers
2k views

elasticsearch using bulk insert to post JSON file

I have the following JSON file I have used awk to get rid of empty spaces, trailing, next line awk -v ORS= -v OFS= '{$1=$1}1' data.json I have added a create request at the top of my data.json ...
3
votes
0answers
1k views

Curl receives empty repsonse form Elasticsearch server

I have a healthy ES cluster running on ec2. I'm tyring to connect Graylog server to the cluster using unicast, but Graylog receives an empty reponse from the unicast host. Using curl, I can ...
3
votes
0answers
2k views

Shutting down elasticsearch completely?

I have found a number of posts about shutting down a node or a cluster of nodes, but what about stopping all elasticsearch processes on a machine? Even after shutting down all nodes (curl -XPOST '...
3
votes
0answers
314 views

elasticsearch fails periodically on ubuntu with systemd : how to resolve this?

I am using Elasticsearch on Ubuntu 14.10 and it's periodically going down (entering a failed state). When down, this throws exceptions on my website, users can't search, connect or update database ...
3
votes
0answers
3k views

rsyslog - How to flush queue?

I'm using rsyslog to get logs from haproxy and put them into logstash for elasticsearch/kibana. Everything was working ok, but i've found something strange in rsyslog. I've found that i've got ...
2
votes
0answers
43 views

AWS ElasticSearch on private subnet - safe to have no signed url/access policy?

I have an ES cluster setup within our VPC and attached to one of our private subnets. It's also attached to a security group allowing access from particular instances. I have the following policy ...
2
votes
0answers
512 views

Can't access ElasticSearch through local port forward

I'm stuck at connecting to my SearchGuard-proteced ElasticSearch instance inside the company network from my local machine at home. I'm using a local SSH forward same as I got working for a phpMyAdmin ...
2
votes
0answers
455 views

Benchmarking for DISK I/O Writes Per Second - ElasticSearch and MySQL

During working on our project we needed to decide should we use "ElasticSearch" or "MySQL" to record the audit/trail logs of the system. Searching speed is not an issue here, we needed to see the disk ...
2
votes
0answers
798 views

ElasticSearch major slowdown upon big aggregations

My goal is to use ElasticSearch v1.3.2 for analyzing product cross-sales, so I need to filter for the receipts of interest (having an alcoholic product for example) and finding top selling products of ...
2
votes
0answers
515 views

How do I enable digest auth for Jetty fronting Elastic Search

We've gotten Jetty running the HTTP front end of Elastic Search with BASIC authentication. We would like to use DIGEST authentication instead, but the Jetty documentation is hard to reconcile with the ...
2
votes
0answers
220 views

Elasticsearch and couchdb river indexing slows down after a few hours

Trying to import ~400m docs in to Elasticsearch from couchdb using the couchdb river plugin. Everything starts out great with indexing time around 5k/s but after a few hours come back and find its ...
1
vote
0answers
35 views

Parsing JSON event in Logstash

I have log in following format, it is a plain json with nested fields. { "level": "info", "message": { "req": { "headers": { "host": "localhost:8080", ...
1
vote
0answers
28 views

ElasticSearch not starting properly

On startup, our ElasticSearch server throws aJava exception I can see in the elasticsearch.log: [2019-02-27T15:51:59,908][WARN ][o.e.m.j.JvmGcMonitorService] [KibjtLy] [gc][young][263][19] duration [...
1
vote
0answers
31 views

VPC subnet error when deploying Bitname Elasticsearch to non-default GCE network

Deploying Bitnami's single-VM Elasticsearch. The error is: Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network It works fine if ...
1
vote
0answers
214 views

Logstash losing connection to Elasticsearch nodes

We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We ...
1
vote
0answers
200 views

Configure Google Cloud Storage plugin with Curator cronjob on GKE Elasticsearch cluster

I deployed an Elasticsearch cluster on GKE, using this project as a starting point: https://github.com/pires/kubernetes-elasticsearch-cluster I would like to configure two Curator jobs by using ...
1
vote
0answers
35 views

CloudWatch Logs, Elasticsearch service expiration

I'm using CloudWatch Logs streaming to Amazon Elasticsearch service (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html) Currently, I've set Events expiration in Cloudwatch ...
1
vote
0answers
68 views

Elasticsearch: fine-tuning for RAID0 HDD array storage

It is usually recommended to have the following index settings in case spinning disks are used for Elasticsearch data storage: index.store.throttle.type: none index.merge.scheduler.max_thread_count: ...
1
vote
0answers
316 views

elasticsearch - delete specific indices from snapshot

I used Snapshot And Restore | Elasticsearch Reference to setup snapshot, however I did not specify which one of many indices I need to take snapshot of, which lead to 390 (at the moment) unneeded ...
1
vote
0answers
161 views

Why does syslog create a user.log instead using syslog.log?

I have experienced something a bit weird for me. I have filebeat monitoring my rsyslog (syslog.log) file and sending it to my logstash. I have noticed that after restarting filebeat where syslog is ...
1
vote
0answers
107 views

Delayed synchronization between two AWS ElasticSearch Service instances

I need to synchronize the indexes of two AWS ElasticSearch Service instances once a day. It's really important that the synchronization is NOT real time. The first index (let's call it ES-A) is ...
1
vote
0answers
622 views

object mapping for [doc.awsRegion] tried to parse field [awsRegion] as object, but found a concrete value

I am using Logstash Dynamodb PLugin to index data to be able to query it. Everything was running pretty smoothly, until one of my teammates started messing with the schema, and broke it. We can still ...
1
vote
0answers
179 views

Redis creates many events in logstash

I'm doing some testing for a POC with logstash. When using redis to buffer messages it appears they stay in redis and continually get added to elasticsearch as new events. For example, if I restart ...
1
vote
0answers
207 views

Visualize multiline ruby exceptions in kibana 4

I have setup the latest version of Kibana4 ElasticSearch stack. The logs are being pooled from remote app sources which are running on Ruby. I want to search for Multi Line exceptions created by ...
1
vote
0answers
144 views

Filebeat and downstream availability

I read here and there that a broker (like Redis) might not be required in the log pipeline (typically ELK) when Filebeat is used. From Filebeat's official page: [Filebeat] is intelligent enough to ...
1
vote
0answers
23 views

elasticsearch creating files but not releasing them until I restart the service

I am running elastcsearch 0.90.5 and recently I have been having issues with the disk running out of space. This is not caused by my indexes increasing in size but by the elasticsearch service ...
1
vote
0answers
81 views

Elasticsearch shuts down with no reason

We're running Elasticsearch 1.7.2 on ubuntu 14.04, the system has enough memory and other services on the server work just fine. But sometimes elastic shutsdown without any thing logged to it's log, ...
1
vote
0answers
44 views

Experiences with logfile monitoring

In an IT-integrator, I'm currently trying to evangelize the use of log file monitoring of hardware installations. The idea is to use a Elasticsearch/Logstash/Kibana stack (most flexibility, open ...
1
vote
0answers
809 views

How do I update Logstash data to Redshift real-time?

I am looking for a way to build a pipeline for storing the logs data (I use the ELK stack) to AWS Redshift, for analytics. I have gone through the Output plugins for ElasticSearch, and found one for ...
1
vote
0answers
1k views

What is wrong with my logstash configuration?

Background I have two types of log files: output from an ETL process, and then output from a downstream processor. We call these "ETL" and "Processor" logs. The ETL logs are in their own folder ...
1
vote
0answers
725 views

Elasticsearch snapshot / curl error

I'm currently having an issue getting backups for ElasticSearch working correctly via their snapshotting system. Per their instructions, a snapshot repository has to be configured first, using this ...
1
vote
0answers
582 views

elasticsearch glibc memory corruption

My elastic search clusters keep crashing for no reason. All I can see in the logs is : INFO | jvm 4 | 2015/04/17 19:10:09 | *** glibc detected *** /usr/java/jre1.7.0_04/bin/java: malloc(): ...
1
vote
0answers
290 views

How parse a multiple objects serialized in log with logstash and grok

I'm parsing some log files with grok, but I have the following trace: [es.org.some.Object[ attr1=53212 searchTags=[love, rock] ]] [es.org.some.Object[ attr1=54612 searchTags=[love, peace, rock] ...
1
vote
0answers
2k views

logstash / kibana can't connect to instance

I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries. However.. today when I went to my logstash page, I see ...
1
vote
0answers
2k views

JVM process resident set size “equals” max heap size, not current heap size

After a few reading about jvm memory (here, here, here, others I forgot...), I am expecting the resident set size of my java process to be roughly equal to the current heap space capacity. That's not ...
0
votes
0answers
14 views

Elastic search monitoring with graphite grafana (Better solution?)

We have created a grafana dashboard which is configured for Elasticsearch resource monitoring and we are using python script to fetch resource information from ES and put it in to Graphite. We are ...
0
votes
0answers
8 views

Elasticsearch snapshot to azure - option not appearing

I've installed the Azure repository plugin for 6.7, and configured a blob for it to write to. The directions at https://www.elastic.co/guide/en/elasticsearch/plugins/6.7/repository-azure.html have ...
0
votes
0answers
33 views

Getting Apache2 combined logs into the Elastic Stack and using dashboards

I am looking into the ELK stack, and I mostly got it running. My goal would be to get the Apache2 dashboards running in Kibana, but I am unable to. This is how far I got: **On the Apache side:* I am ...
0
votes
0answers
27 views

Moving data to a new elasticsearch index with more shards

We seem to be running out of shards on some of our larger indices on Elasticsearch 1.7. In order to utilize larger clusters with more nodes, say 20 instead of 10, I'm hoping that we can create new ...
0
votes
0answers
80 views

Logstash: Failed to decode CEF payload

I am parsing via the cef input plugin in Logstash log files in cef format to elasticsearch. At some log files I recevice the message Failed to decode CEF payload. It seems that the problem could be ...
0
votes
0answers
35 views

Filebeat is not sending data to Logstash

I’m trying to send alerts from Snort IDS to Elasticsearch, therefore I'm using 3 technologies: Elasticsearch- https://pastebin.com/uCNMaZFJ Logstash-https://pastebin.com/zgnbbw9K Filebeat- https://...
0
votes
0answers
73 views

fluentd error parsing auth.log

I have fluentd setup for Kubernetes which is working just fine but when add more sources it fails. I'm trying to get auth.log like this: <source> @id authlog @type tail format none ...
0
votes
0answers
17 views

Migrating from a public elastic cluster to VPC cluster

I currently, have one public elastic cluster with one data node(m3.xlarge.elasticsearch) which is working fine. I would like to move from the public domain to VPC based domain. I was able to take a ...
0
votes
0answers
92 views

Logstash isn't accepting the self signed certificates and throws a sslv3 alert handshake failure

I have set up an elsatic stack in my laptop and I have configured full stack (filebeat, logstash, elasticsearch and kibana) in it. But now I need to connect filebeat and logstash in a secure manner ...
0
votes
0answers
29 views

AWS Elasticsearch access policy with Kinesis and EC2

I am currently working on an AWS setup where data is pushed to an Elasticsearch service via Amazon Kinesis (Firehose). This data has to be accessible from all EC2 instances that are connected to a ...
0
votes
0answers
156 views

Ceph vs GlusterFS for elasticsearch storage

Elasticsearch will be holding logs mostly, so we expect sequential read/writes. Does anyone have experience on this? Elasticsearch will run via openshift, if it makes any difference.
0
votes
0answers
120 views

Multiple multiline parsers in Kubernetes Filebeat config

I'm using Filebeat to ship logs to Kubernetes. How can I configure multiline collation differently based on different conditions? E.g., use a Java log regex for my Java containers, and a PHP regex for ...
0
votes
0answers
225 views

Running Sonarqube, elasticsearch throws error in sonar.log

Installed and started Sonarqube up, and in the sonar.log it complains about access denied for elasticsearch.yml. It's surely permissions, but I can't figure out what exactly. Sonarqube 7.3 and ...
0
votes
0answers
23 views

fluent and parsing of logs

I'm trying to centralize logs using fluent. That part seems to be ok, I have a central fluent and other nodes and I use forward plugin to push them. Once on the central fluent I have 2 outputs, one ...
0
votes
0answers
39 views

ElasticSearch make replica shard primary

We have a small ElasticSearch 2.4 cluster containing 2 Data Nodes and multiple Indicies. The indicies have 2 primary shards and 1 replica each. If we create a new index using our default template, ...