Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

18
votes
1answer
53k views

How to check ElasticSearch version in Linux [closed]

I am just curious on how to check ElasticSearch version in Linux?
16
votes
1answer
14k views

Scaling Logstash (with redis/elasticsearch)

Over a cluster of over 12 centos 5.8 servers, I deployed logstash using the native logstash shipper, which sends /var/log/*/*.log back to a central logstash server. We tried using rsyslogd as the ...
13
votes
3answers
24k views

Unable to change vm.max_map_count for elasticsearch

Prehistory I have elasticsearch and SugarCRM7 running on CentOS 6.5. Every day I face the same problem: java outOfMemory error. That happens because of small vm.max_map_count value, 65530 only when ...
12
votes
4answers
54k views

Ubuntu uninstall elasticsearch

I installed elasticsearch.90.7 with a deb file in ubuntu. I tried to uninstall elasticsearch.90.7 with this command: sudo apt-get --purge autoremove elasticsearch And then I downloaded elasticsearch-...
10
votes
1answer
32k views

Elasticsearch is using way too much disk space

I have a CentOS 6.5 server on which I installed Elasticsearch 1.3.2. My elasticsearch.yml configuration file is a minimal modification of the one shipping with elasticsearch as a default. Once ...
10
votes
1answer
7k views

Recommendations for Elastic Search hardware [closed]

Are there any good guides for hardware level to support ElasticSearch? Are recommendations for Lucene or Solr a good place to start? We're looking at rolling out a deployment starting with 27 million ...
9
votes
1answer
423 views

Elasticsearch dies when Logstash attempts to write data

I've got a Raspberry Pi 2 (latest Raspbian as of Apr 2015) setup that last week was running both ElasticSearch and Logstash on a test network (not a straightforward setup, but it was stable for over a ...
8
votes
4answers
4k views

How best to monitor logstash?

I've seen this question on the mailing list a few times but haven't had a satisfactory answer. How best to monitor that the pipeline isn't stuck? Clients -> logstash -> elasticsearch. Logstash and ...
7
votes
3answers
7k views

Secure logstash and elasticsearch

I'm considering running logstash on my prod server (simple install. http://logstash.net/docs/1.1.13/tutorials/getting-started-simple) and set kibana to access logs. My concern is: how to secure my ...
7
votes
4answers
23k views

Error Could not contact Elasticsearch at http://localhost:9200. - kibana dashboard

'Error Could not contact Elasticsearch at http://localhost:9200. Please ensure that Elasticsearch is reachable from your system.' Currently I am following http://packetbeat.com/getstarted I ...
6
votes
1answer
9k views

How well does elasticsearch compress data?

I am looking to scope the servers required for an elasticsearch proof-of-concept. Ultimately, my question is this: Given 1GB of json text indexed by elasticsearch, how much can I expect to be ...
6
votes
3answers
6k views

Elastic Search listening only on IPv6 [CentOS]

I've installed Elastic Search 1.7.4 on CentOS 7.2 this way: wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.4.noarch.rpm sudo rpm -ivh elasticsearch-1.7.4.noarch.rpm ...
6
votes
1answer
10k views

ElasticSearch Server Randomly Stops Working

I have 2 ES servers that are being fed by 1 logstash server and viewing the logs in Kibana. This is a POC to work out any issues before going into production. The system has ran for ~1 month and every ...
6
votes
0answers
347 views

Is there disk based file system with dedicated memory caching for Linux? [closed]

We have two (or more sometimes) applications battling for the OS page cache. In particular ElasticSearch is very dependent on the OS page cache. To the point that cache misses can make or break ...
5
votes
1answer
1k views

Why would a node not have shards?

I am setting up a small elasticsearch cluster and one of the nodes does not get any shards. It works, is part of the cluster - just empty (eu2 in the screenshot of head below): This node is a worker, ...
5
votes
3answers
9k views

Ubuntu 15.04: Elasticsearch service doesn't start

I followed the first part from the tutorial here to install Elasticsearch. I didn't install java because I already have that installed with other java programs running. sudo service elasticsearch ...
5
votes
2answers
6k views

How to index MySQL from ElasticSearch

Im trying to add some full text searching to my LAMP web application. Ive come across ElasticSearch and I like what it has to offer. I have read the guides and understand how to index a single item by ...
5
votes
2answers
2k views

How to secure an Internet-facing Elastic Search implementation in a shared hosting environment?

(Originally asked on StackOverflow, and recommended that I move it here) I've been going over the documentation for Elastic Search and I'm a big fan and I'd like to use it to handle the search for my ...
5
votes
1answer
294 views

Different linux page cache behavior for servers doing the same work

I have two sets of servers w/128G memory, distinguished by when they were provisioned, that are behaving very differently while running the exact same daemon (elasticsearch). I am using elasticsearch ...
5
votes
2answers
1k views

Sending Subversion logs to Logstash

My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own ...
5
votes
2answers
3k views

“Index Patterns: Please specify a default index pattern” in Kibana

I'm trying to create a simple hello world for ELK and be able to see kibana reports via the internet. I've installed kibana, logstash, nginx and elastic search. Here's my /etc/logstash/conf.d/10-...
5
votes
1answer
609 views

Elasticsearch Debugging

Our elasticsearch is a mess. The cluster health is always in red and ive decided to look into it and salvage it if possible. But I have no idea where to begin with. Here is some info regarding our ...
4
votes
4answers
31k views

Nothing appearing in kibana dashboard

I have installed Logstash ElasticSearch Kibana on an EC2 instance. I can visit http://example.com:9200 Here I get { "status" : 200, "name" : "Aleta Ogord", "version" : { "number" : "1.1.0"...
4
votes
1answer
618 views

ElasticSearch snapshot backup sliding expiration - possible?

I'm planning on using the ElasticSearch S3 cloud plugin to create snapshots of our ES cluster. This all looks fairly straight forward but I'm wondering whether its possible integrate it into our ...
4
votes
4answers
532 views

Securing elasticsearch

There appears to be a serious lack of information on this despite the fact that as installed elasticsearch is extremely exploitable. My main fear in using it is that as a non-expert I have no idea ...
4
votes
1answer
3k views

Adding TTL to elasticsearch with logstash

I can't figure out how to apply this documentation http://www.elasticsearch.org/guide/reference/mapping/ttl-field/ to logstash. Specifically, i don't know which keys to use (instead of "tweet" etc). ...
4
votes
1answer
2k views

Logstash/elasticsearch stops accepting new data

I've set up a new proof of concept logstash system CentOS 6.6 (on Vmware 5.5) - single CPU VM with 12G RAM allocated Elasticsearch and Logstash installed from RPMs … # rpm -q elasticsearch logstash ...
4
votes
1answer
1k views

Elasticsearch: High traffic on loopback interface

One of our application uses Elasticsearch (1.4.4) as an in-memory cache. The application is a Java webapp deployed on Tomcat 7 with Oracle 1.7. The elasticsearch instance is a one-node setup deployed ...
4
votes
1answer
136 views

elasticsearch server is unreachable every 2 hours

This question is related to this one. We now know that the errors come from elasticsearch. The problems are still not resolved despite the modifications and optimizations made on the es instance. ...
4
votes
0answers
2k views

elasticsearch using bulk insert to post JSON file

I have the following JSON file I have used awk to get rid of empty spaces, trailing, next line awk -v ORS= -v OFS= '{$1=$1}1' data.json I have added a create request at the top of my data.json ...
4
votes
2answers
2k views

Elasticsearch performance tuning

In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge. Logstash conf ...
4
votes
3answers
8k views

Foward slash in kibana 3 query

I'm trying to add a query that will match a request that ends with a slash, like this one: n.n.n.n - - [16/Oct/2013:16:40:41 +0100] "GET / HTTP/1.1" 200 25058 "-" "Mozilla/5.0 (iPad; CPU OS 7_0_2 ...
3
votes
3answers
423 views

Simple full-text search server

I have been looking at search solutions like Sphinx, Solr and Elasticsearch but they are all way too complex for what I need. I'm basically looking for a server software, best distributed, that ...
3
votes
3answers
1k views

Must all nodes of an Elasticsearch cluster be identical?

In an Elasticsearch cluster, must all nodes be identical i.e. having the same: Operating System Amount of RAM CPU speed and number of cores etc. or is it allowed for the nodes to be different? (...
3
votes
2answers
8k views

No route to host elasticsearch exception

When turning on elasticsearch, I am getting the following configuration error: [2015-04-21 20:49:45,635][INFO ][discovery.zen ] [Blackwulf] failed to send join request to master [[...
3
votes
2answers
6k views

What is the purpose of elasticsearch logs? How to manage them?

As I understand indexes(or data) are being stored in /var/lib/elasticsearch by default, this folder contains nodes with 0 and 1 folders and overall size of these folders is 376M. The logs are ...
3
votes
1answer
2k views

Migrating to AWS Cloud with auto-scaling - where to put Redis and ElasticSearch?

I've been trying to research this topic but haven't found anywhere that recommends where to install services such as Redis and ElasticSearch when migrating to a cloud framework. I'm currently running ...
3
votes
1answer
4k views

Is there a way to stop the AWS Elasticsearch?

I setup an AWS Elasticsearch Domain recently but I didn't see a way to stop it (like you can with an EC2 instance), which means I'm continuously billed. At this stage I just need to do some testing ...
3
votes
1answer
2k views

Logstash integration with AWS Elasticsearch Service

I am using AWS Elasticsearch service to configure Elasticsearch Cluster and there is a separate server where I have installed Logstash 2.1.0 Here is my Logstash sample configuration file :- input { ...
3
votes
2answers
1k views

Full data backup to Amazon S3

I have an Ubuntu server hosted on Digital Ocean that has outgrown it's existing backup solution. The relevant parts of the stack I use are Node.js, MongoDB, and Elasticsearch. So far backups have ...
3
votes
1answer
3k views

How to rename and join an elasticsearch server preserving data?

I tested elasticsearch and decided to make a cluster. To this I renamed the node and defined a cluster cluster.name = mycluster node.name = firstnode When restarting the changes are taken into ...
3
votes
1answer
864 views

Why is this exclude_lines in filebeat excluding all logs?

I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-...
3
votes
1answer
11k views

elk stack error “unable to fetch mapping do you have indices matching the pattern”

i am trying to setup ELK stack with collectd on Ubuntu 16.04 LTS (so pretty much latest version of stack available) kibana is behind nginx proxy (followed this guide https://www.digitalocean.com/...
3
votes
1answer
3k views

Using Logstash as shipper?

We are shipping logs from servers and using THE logstash on each server for shipping. So we read logs from the glob "/root/Desktop/Logstash-Input/**/*_log". input { file{ ...
3
votes
1answer
5k views

Logstash Server Log FIle Flooded With “Failed to flush outgoing items” Error Messages

I'm new to Logstash, trying to get it running following the tutorial in "The Logstash Book". On page 44, the guide suggests to tail the Logstash process' main log file called central.log. After ...
3
votes
1answer
2k views

New ElasticSearch 5.2.1 Install on Ubuntu 16.04: Connection refused

I just followed this install guide for Elasticsearch via APT: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/deb.html When I run curl -X GET 'http://127.0.0.1:9200/' (or localhost) I ...
3
votes
1answer
4k views

Large virtual memory size of ElasticSearch JVM

I am running a JVM to support ElasticSearch. I am still working on sizing and tuning, so I left the JVM's max heap size at ElasticSearch's default of 1GB. After putting data in the database, I find ...
3
votes
1answer
448 views

Logstash - Graph IPv4 vs IPv6

I'm looking at starting to graph inbound connections to our various services (web, mail, dns etc) by IP version, ie IPv4 connections versus IPv6 connections. Currently the IPs are being stored in a ...
3
votes
0answers
1k views

Curl receives empty repsonse form Elasticsearch server

I have a healthy ES cluster running on ec2. I'm tyring to connect Graylog server to the cluster using unicast, but Graylog receives an empty reponse from the unicast host. Using curl, I can ...
3
votes
0answers
2k views

Shutting down elasticsearch completely?

I have found a number of posts about shutting down a node or a cluster of nodes, but what about stopping all elasticsearch processes on a machine? Even after shutting down all nodes (curl -XPOST '...