Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

4
votes
0answers
2k views

elasticsearch using bulk insert to post JSON file

I have the following JSON file I have used awk to get rid of empty spaces, trailing, next line awk -v ORS= -v OFS= '{$1=$1}1' data.json I have added a create request at the top of my data.json ...
3
votes
0answers
1k views

Curl receives empty repsonse form Elasticsearch server

I have a healthy ES cluster running on ec2. I'm tyring to connect Graylog server to the cluster using unicast, but Graylog receives an empty reponse from the unicast host. Using curl, I can ...
3
votes
0answers
2k views

Shutting down elasticsearch completely?

I have found a number of posts about shutting down a node or a cluster of nodes, but what about stopping all elasticsearch processes on a machine? Even after shutting down all nodes (curl -XPOST '...
3
votes
0answers
316 views

elasticsearch fails periodically on ubuntu with systemd : how to resolve this?

I am using Elasticsearch on Ubuntu 14.10 and it's periodically going down (entering a failed state). When down, this throws exceptions on my website, users can't search, connect or update database ...
3
votes
0answers
3k views

rsyslog - How to flush queue?

I'm using rsyslog to get logs from haproxy and put them into logstash for elasticsearch/kibana. Everything was working ok, but i've found something strange in rsyslog. I've found that i've got ...
2
votes
0answers
424 views

Parsing JSON event in Logstash

I have log in following format, it is a plain json with nested fields. { "level": "info", "message": { "req": { "headers": { "host": "localhost:8080", ...
2
votes
0answers
60 views

AWS ElasticSearch on private subnet - safe to have no signed url/access policy?

I have an ES cluster setup within our VPC and attached to one of our private subnets. It's also attached to a security group allowing access from particular instances. I have the following policy ...
2
votes
0answers
555 views

Can't access ElasticSearch through local port forward

I'm stuck at connecting to my SearchGuard-proteced ElasticSearch instance inside the company network from my local machine at home. I'm using a local SSH forward same as I got working for a phpMyAdmin ...
2
votes
0answers
464 views

Benchmarking for DISK I/O Writes Per Second - ElasticSearch and MySQL

During working on our project we needed to decide should we use "ElasticSearch" or "MySQL" to record the audit/trail logs of the system. Searching speed is not an issue here, we needed to see the disk ...
2
votes
0answers
802 views

ElasticSearch major slowdown upon big aggregations

My goal is to use ElasticSearch v1.3.2 for analyzing product cross-sales, so I need to filter for the receipts of interest (having an alcoholic product for example) and finding top selling products of ...
2
votes
0answers
519 views

How do I enable digest auth for Jetty fronting Elastic Search

We've gotten Jetty running the HTTP front end of Elastic Search with BASIC authentication. We would like to use DIGEST authentication instead, but the Jetty documentation is hard to reconcile with the ...
2
votes
0answers
221 views

Elasticsearch and couchdb river indexing slows down after a few hours

Trying to import ~400m docs in to Elasticsearch from couchdb using the couchdb river plugin. Everything starts out great with indexing time around 5k/s but after a few hours come back and find its ...
1
vote
0answers
19 views

What is causing the Elasticsearch service to ignore my jvm.options?

In my /etc/elasticsearch/jvm.options I have set my heap size with: -Xms256m -Xmx256m However when I restart the service, memory keeps rising until it runs out and the service is stopped abruptly: [...
1
vote
0answers
23 views

AWS : Replicate Amazon Elastic search and RDS

We are working on an AWS system where we are using Amazon RDS for database purposes and ElasticSearch for indexing. We have a requirement that we require weekly backup of both the services along with ...
1
vote
0answers
54 views

ElasticSearch not starting properly

On startup, our ElasticSearch server throws aJava exception I can see in the elasticsearch.log: [2019-02-27T15:51:59,908][WARN ][o.e.m.j.JvmGcMonitorService] [KibjtLy] [gc][young][263][19] duration [...
1
vote
0answers
34 views

VPC subnet error when deploying Bitname Elasticsearch to non-default GCE network

Deploying Bitnami's single-VM Elasticsearch. The error is: Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network It works fine if ...
1
vote
0answers
282 views

Logstash losing connection to Elasticsearch nodes

We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We ...
1
vote
0answers
233 views

Configure Google Cloud Storage plugin with Curator cronjob on GKE Elasticsearch cluster

I deployed an Elasticsearch cluster on GKE, using this project as a starting point: https://github.com/pires/kubernetes-elasticsearch-cluster I would like to configure two Curator jobs by using ...
1
vote
0answers
41 views

CloudWatch Logs, Elasticsearch service expiration

I'm using CloudWatch Logs streaming to Amazon Elasticsearch service (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html) Currently, I've set Events expiration in Cloudwatch ...
1
vote
0answers
70 views

Elasticsearch: fine-tuning for RAID0 HDD array storage

It is usually recommended to have the following index settings in case spinning disks are used for Elasticsearch data storage: index.store.throttle.type: none index.merge.scheduler.max_thread_count: ...
1
vote
0answers
340 views

elasticsearch - delete specific indices from snapshot

I used Snapshot And Restore | Elasticsearch Reference to setup snapshot, however I did not specify which one of many indices I need to take snapshot of, which lead to 390 (at the moment) unneeded ...
1
vote
0answers
189 views

Why does syslog create a user.log instead using syslog.log?

I have experienced something a bit weird for me. I have filebeat monitoring my rsyslog (syslog.log) file and sending it to my logstash. I have noticed that after restarting filebeat where syslog is ...
1
vote
0answers
110 views

Delayed synchronization between two AWS ElasticSearch Service instances

I need to synchronize the indexes of two AWS ElasticSearch Service instances once a day. It's really important that the synchronization is NOT real time. The first index (let's call it ES-A) is ...
1
vote
0answers
648 views

object mapping for [doc.awsRegion] tried to parse field [awsRegion] as object, but found a concrete value

I am using Logstash Dynamodb PLugin to index data to be able to query it. Everything was running pretty smoothly, until one of my teammates started messing with the schema, and broke it. We can still ...
1
vote
0answers
180 views

Redis creates many events in logstash

I'm doing some testing for a POC with logstash. When using redis to buffer messages it appears they stay in redis and continually get added to elasticsearch as new events. For example, if I restart ...
1
vote
0answers
214 views

Visualize multiline ruby exceptions in kibana 4

I have setup the latest version of Kibana4 ElasticSearch stack. The logs are being pooled from remote app sources which are running on Ruby. I want to search for Multi Line exceptions created by ...
1
vote
0answers
147 views

Filebeat and downstream availability

I read here and there that a broker (like Redis) might not be required in the log pipeline (typically ELK) when Filebeat is used. From Filebeat's official page: [Filebeat] is intelligent enough to ...
1
vote
0answers
23 views

elasticsearch creating files but not releasing them until I restart the service

I am running elastcsearch 0.90.5 and recently I have been having issues with the disk running out of space. This is not caused by my indexes increasing in size but by the elasticsearch service ...
1
vote
0answers
83 views

Elasticsearch shuts down with no reason

We're running Elasticsearch 1.7.2 on ubuntu 14.04, the system has enough memory and other services on the server work just fine. But sometimes elastic shutsdown without any thing logged to it's log, ...
1
vote
0answers
44 views

Experiences with logfile monitoring

In an IT-integrator, I'm currently trying to evangelize the use of log file monitoring of hardware installations. The idea is to use a Elasticsearch/Logstash/Kibana stack (most flexibility, open ...
1
vote
0answers
826 views

How do I update Logstash data to Redshift real-time?

I am looking for a way to build a pipeline for storing the logs data (I use the ELK stack) to AWS Redshift, for analytics. I have gone through the Output plugins for ElasticSearch, and found one for ...
1
vote
0answers
1k views

What is wrong with my logstash configuration?

Background I have two types of log files: output from an ETL process, and then output from a downstream processor. We call these "ETL" and "Processor" logs. The ETL logs are in their own folder ...
1
vote
0answers
734 views

Elasticsearch snapshot / curl error

I'm currently having an issue getting backups for ElasticSearch working correctly via their snapshotting system. Per their instructions, a snapshot repository has to be configured first, using this ...
1
vote
0answers
588 views

elasticsearch glibc memory corruption

My elastic search clusters keep crashing for no reason. All I can see in the logs is : INFO | jvm 4 | 2015/04/17 19:10:09 | *** glibc detected *** /usr/java/jre1.7.0_04/bin/java: malloc(): ...
1
vote
0answers
300 views

How parse a multiple objects serialized in log with logstash and grok

I'm parsing some log files with grok, but I have the following trace: [es.org.some.Object[ attr1=53212 searchTags=[love, rock] ]] [es.org.some.Object[ attr1=54612 searchTags=[love, peace, rock] ...
1
vote
0answers
2k views

logstash / kibana can't connect to instance

I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries. However.. today when I went to my logstash page, I see ...
1
vote
0answers
2k views

JVM process resident set size “equals” max heap size, not current heap size

After a few reading about jvm memory (here, here, here, others I forgot...), I am expecting the resident set size of my java process to be roughly equal to the current heap space capacity. That's not ...
0
votes
0answers
21 views

Creating email alerts for kibana filters via sentil

I have a elk stack that works properly and can be monitored via the kibana dashboard. But as an improvement to the whole setup I have being researching for a reporting tool to setup with kibana. My ...
0
votes
0answers
8 views

Determine running version of elasticsearch, without using http interface

If an elasticsearch node supports the http interface, you can check the running version via curl -XGET 'localhost:9200' (or whatever port the http interface is on). If you want to know the installed ...
0
votes
0answers
17 views

logstash not able to upload data to elasticsearch even the pipeline started

I am using elasticsearch7.1.1 and logstash7.1.1 . I am trying to uplog a log file to elastic search using grok filter. pipeline is getting started but data is not getting uploaded. here is my config ...
0
votes
0answers
9 views

Read-only Access to Kibana/ESAAS

Allowing "GET" Operations only via a group is breaking kibana access. When user accesses kibana with credentials; the "discover" page is not displayed. The page "Localhost:5601/app/kibana#/discover?_g"...
0
votes
0answers
37 views

SonarQube 7.7 Does not start on CentOS Linux

Title says everything. Same error happens with SonarQube 6.7.7 LTS but I write only 7.7 issues here. I have no version restriction which I should use. I downloaded SonarQube 7.7 from your website. ...
0
votes
0answers
14 views

Elasticsearch offline-archive and restore

How Elasticsearch can use 'archive'. On archive I mean logs stored offline after a specified retention time.  Is it possible to achive? How can it be done? How the offline indexes can be restored ...
0
votes
0answers
29 views

Elasticsearch connection refused

I have just installed ElasticSearch 7.1.1 on Debian 9 throw apt-get repository VPS 4GB ram .. 1vcpu service elasticsearch status elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/...
0
votes
0answers
29 views

Elasticsearch does not start on Ubuntu 14.04, possible error with lucene dependency

I have tried installing the elasticsearch package using apt on Ubuntu 14.04, but when I run sudo service elasticsearch status, I get the following output: * elasticsearch is not running I tried ...
0
votes
0answers
20 views

Elastic search monitoring with graphite grafana (Better solution?)

We have created a grafana dashboard which is configured for Elasticsearch resource monitoring and we are using python script to fetch resource information from ES and put it in to Graphite. We are ...
0
votes
0answers
15 views

Elasticsearch snapshot to azure - option not appearing

I've installed the Azure repository plugin for 6.7, and configured a blob for it to write to. The directions at https://www.elastic.co/guide/en/elasticsearch/plugins/6.7/repository-azure.html have ...
0
votes
0answers
119 views

Getting Apache2 combined logs into the Elastic Stack and using dashboards

I am looking into the ELK stack, and I mostly got it running. My goal would be to get the Apache2 dashboards running in Kibana, but I am unable to. This is how far I got: **On the Apache side:* I am ...
0
votes
0answers
37 views

Moving data to a new elasticsearch index with more shards

We seem to be running out of shards on some of our larger indices on Elasticsearch 1.7. In order to utilize larger clusters with more nodes, say 20 instead of 10, I'm hoping that we can create new ...
0
votes
0answers
188 views

Logstash: Failed to decode CEF payload

I am parsing via the cef input plugin in Logstash log files in cef format to elasticsearch. At some log files I recevice the message Failed to decode CEF payload. It seems that the problem could be ...