Questions tagged [elasticsearch]
Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.
380
questions
37
votes
6
answers
54k
views
How to increase disk size in a stateful set
I'm managing an Elasticsearch deployment in Kubernetes. I see that the disk storage is getting close to full, so I would like to increase the size of the persistent volumes.
I want to change this ...
24
votes
1
answer
79k
views
How to check ElasticSearch version in Linux [closed]
I am just curious on how to check ElasticSearch version in Linux?
20
votes
4
answers
80k
views
Ubuntu uninstall elasticsearch
I installed elasticsearch.90.7 with a deb file in ubuntu.
I tried to uninstall elasticsearch.90.7 with this command:
sudo apt-get --purge autoremove elasticsearch
And then I downloaded elasticsearch-...
18
votes
2
answers
21k
views
Is there a way to stop the AWS Elasticsearch?
I setup an AWS Elasticsearch Domain recently but I didn't see a way to stop it (like you can with an EC2 instance), which means I'm continuously billed. At this stage I just need to do some testing ...
16
votes
1
answer
14k
views
Scaling Logstash (with redis/elasticsearch)
Over a cluster of over 12 centos 5.8 servers, I deployed logstash using the native logstash shipper, which sends /var/log/*/*.log back to a central logstash server.
We tried using rsyslogd as the ...
16
votes
3
answers
40k
views
Unable to change vm.max_map_count for elasticsearch
Prehistory
I have elasticsearch and SugarCRM7 running on CentOS 6.5. Every day I face the same problem: java outOfMemory error. That happens because of small vm.max_map_count value, 65530 only when ...
13
votes
1
answer
37k
views
Elasticsearch is using way too much disk space
I have a CentOS 6.5 server on which I installed Elasticsearch 1.3.2.
My elasticsearch.yml configuration file is a minimal modification of the one shipping with elasticsearch as a default. Once ...
10
votes
1
answer
7k
views
Recommendations for Elastic Search hardware [closed]
Are there any good guides for hardware level to support ElasticSearch? Are recommendations for Lucene or Solr a good place to start? We're looking at rolling out a deployment starting with
27 million ...
9
votes
1
answer
485
views
Elasticsearch dies when Logstash attempts to write data
I've got a Raspberry Pi 2 (latest Raspbian as of Apr 2015) setup that last week was running both ElasticSearch and Logstash on a test network (not a straightforward setup, but it was stable for over a ...
8
votes
4
answers
5k
views
How best to monitor logstash?
I've seen this question on the mailing list a few times but haven't had a satisfactory answer.
How best to monitor that the pipeline isn't stuck? Clients -> logstash -> elasticsearch.
Logstash and ...
7
votes
3
answers
16k
views
Elastic Search listening only on IPv6 [CentOS]
I've installed Elastic Search 1.7.4 on CentOS 7.2 this way:
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.4.noarch.rpm
sudo rpm -ivh elasticsearch-1.7.4.noarch.rpm
...
7
votes
2
answers
22k
views
How to install Elasticsearch plugins with docker container?
I am a newbie in Docker, I want to install plugins in my container of elasticsearch, in this case they are:
analysis-icu
analysis-phonetic
I know that in a traditional way it would be like this:
/...
7
votes
3
answers
7k
views
Secure logstash and elasticsearch
I'm considering running logstash on my prod server (simple install. http://logstash.net/docs/1.1.13/tutorials/getting-started-simple) and set kibana to access logs.
My concern is: how to secure my ...
7
votes
4
answers
26k
views
Error Could not contact Elasticsearch at http://localhost:9200. - kibana dashboard
'Error Could not contact Elasticsearch at http://localhost:9200. Please ensure that Elasticsearch is reachable from your system.'
Currently I am following http://packetbeat.com/getstarted
I ...
6
votes
1
answer
10k
views
How well does elasticsearch compress data?
I am looking to scope the servers required for an elasticsearch proof-of-concept.
Ultimately, my question is this:
Given 1GB of json text indexed by elasticsearch, how much can I expect to be ...
6
votes
1
answer
670
views
Different linux page cache behavior for servers doing the same work
I have two sets of servers w/128G memory, distinguished by when they were provisioned, that are behaving very differently while running the exact same daemon (elasticsearch). I am using elasticsearch ...
6
votes
2
answers
1k
views
Sending Subversion logs to Logstash
My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own ...
6
votes
1
answer
11k
views
ElasticSearch Server Randomly Stops Working
I have 2 ES servers that are being fed by 1 logstash server and viewing the logs in Kibana. This is a POC to work out any issues before going into production. The system has ran for ~1 month and every ...
6
votes
0
answers
409
views
Is there disk based file system with dedicated memory caching for Linux? [closed]
We have two (or more sometimes) applications battling for the OS page cache. In particular ElasticSearch is very dependent on the OS page cache. To the point that cache misses can make or break ...
5
votes
2
answers
7k
views
What is the purpose of elasticsearch logs? How to manage them?
As I understand indexes(or data) are being stored in
/var/lib/elasticsearch
by default, this folder contains nodes with 0 and 1 folders and overall size of these folders is 376M.
The logs are ...
5
votes
1
answer
2k
views
Why would a node not have shards?
I am setting up a small elasticsearch cluster and one of the nodes does not get any shards. It works, is part of the cluster - just empty (eu2 in the screenshot of head below):
This node is a worker, ...
5
votes
3
answers
12k
views
Ubuntu 15.04: Elasticsearch service doesn't start
I followed the first part from the tutorial here to install Elasticsearch. I didn't install java because I already have that installed with other java programs running.
sudo service elasticsearch ...
5
votes
2
answers
6k
views
How to index MySQL from ElasticSearch
Im trying to add some full text searching to my LAMP web application. Ive come across ElasticSearch and I like what it has to offer. I have read the guides and understand how to index a single item by ...
5
votes
1
answer
27k
views
What does "GC (Allocation Failure)" mean in my ElasticSearch 5.6 logs?
I see these constantly in my Elasticsearch 5.6.3 logs. Is this a signal that I should scale up and add more RAM? Or is this just normal operations for ElasticSearch?
[GC (Allocation Failure) [...
5
votes
2
answers
2k
views
How to secure an Internet-facing Elastic Search implementation in a shared hosting environment?
(Originally asked on StackOverflow, and recommended that I move it here)
I've been going over the documentation for Elastic Search and I'm a big fan and I'd like to use it to handle the search for my ...
5
votes
2
answers
5k
views
"Index Patterns: Please specify a default index pattern" in Kibana
I'm trying to create a simple hello world for ELK and be able to see kibana reports via the internet. I've installed kibana, logstash, nginx and elastic search. Here's my /etc/logstash/conf.d/10-...
5
votes
0
answers
3k
views
elasticsearch using bulk insert to post JSON file
I have the following JSON file
I have used awk to get rid of empty spaces, trailing, next line
awk -v ORS= -v OFS= '{$1=$1}1' data.json
I have added a create request at the top of my data.json ...
5
votes
1
answer
780
views
Elasticsearch Debugging
Our elasticsearch is a mess. The cluster health is always in red and ive decided to look into it and salvage it if possible. But I have no idea where to begin with. Here is some info regarding our ...
4
votes
4
answers
32k
views
Nothing appearing in kibana dashboard
I have installed
Logstash
ElasticSearch
Kibana
on an EC2 instance.
I can visit http://example.com:9200 Here I get
{
"status" : 200,
"name" : "Aleta Ogord",
"version" : {
"number" : "1.1.0"...
4
votes
3
answers
9k
views
Logstash / Elasticsearch - trasform fields to lowercase in output conf
I have a standard ELK stack currently storing numerous log outputs. I'm trying to separate my indices to be source-specific.
As part of my FileBeats config, some standard fields are always generated ...
4
votes
1
answer
889
views
ElasticSearch snapshot backup sliding expiration - possible?
I'm planning on using the ElasticSearch S3 cloud plugin to create snapshots of our ES cluster. This all looks fairly straight forward but I'm wondering whether its possible integrate it into our ...
4
votes
4
answers
546
views
Securing elasticsearch
There appears to be a serious lack of information on this despite the fact that as installed elasticsearch is extremely exploitable.
My main fear in using it is that as a non-expert I have no idea ...
4
votes
1
answer
3k
views
Adding TTL to elasticsearch with logstash
I can't figure out how to apply this documentation http://www.elasticsearch.org/guide/reference/mapping/ttl-field/ to logstash. Specifically, i don't know which keys to use (instead of "tweet" etc).
...
4
votes
1
answer
2k
views
Logstash/elasticsearch stops accepting new data
I've set up a new proof of concept logstash system
CentOS 6.6 (on Vmware 5.5) - single CPU VM with 12G RAM allocated
Elasticsearch and Logstash installed from RPMs …
# rpm -q elasticsearch logstash
...
4
votes
1
answer
2k
views
Elasticsearch: High traffic on loopback interface
One of our application uses Elasticsearch (1.4.4) as an in-memory cache. The application is a Java webapp deployed on Tomcat 7 with Oracle 1.7. The elasticsearch instance is a one-node setup deployed ...
4
votes
1
answer
248
views
elasticsearch server is unreachable every 2 hours
This question is related to this one. We now know that the errors come from elasticsearch. The problems are still not resolved despite the modifications and optimizations made on the es instance. ...
4
votes
2
answers
3k
views
Elasticsearch performance tuning
In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge.
Logstash conf
...
4
votes
3
answers
10k
views
Foward slash in kibana 3 query
I'm trying to add a query that will match a request that ends with a slash, like this one:
n.n.n.n - - [16/Oct/2013:16:40:41 +0100] "GET / HTTP/1.1" 200 25058 "-" "Mozilla/5.0 (iPad; CPU OS 7_0_2 ...
3
votes
3
answers
470
views
Simple full-text search server
I have been looking at search solutions like Sphinx, Solr and Elasticsearch but they are all way too complex for what I need.
I'm basically looking for a server software, best distributed, that ...
3
votes
3
answers
3k
views
Must all nodes of an Elasticsearch cluster be identical?
In an Elasticsearch cluster, must all nodes be identical i.e. having the same:
Operating System
Amount of RAM
CPU speed and number of cores
etc.
or is it allowed for the nodes to be different? (...
3
votes
2
answers
11k
views
No route to host elasticsearch exception
When turning on elasticsearch, I am getting the following configuration error:
[2015-04-21 20:49:45,635][INFO ][discovery.zen ] [Blackwulf] failed to send join request to master [[...
3
votes
1
answer
2k
views
Migrating to AWS Cloud with auto-scaling - where to put Redis and ElasticSearch?
I've been trying to research this topic but haven't found anywhere that recommends where to install services such as Redis and ElasticSearch when migrating to a cloud framework.
I'm currently running ...
3
votes
1
answer
7k
views
vm.max_map_count problems on GKE ElasticSearch StatefulSet
A problem appeared on working ElasticSearch cluster on GKE.
Nodes with "data" roles began to crash unexpectedly with an error:
max virtual memory areas vm.max_map_count [65530] is too low, increase ...
3
votes
1
answer
3k
views
Logstash integration with AWS Elasticsearch Service
I am using AWS Elasticsearch service to configure Elasticsearch Cluster and there is a separate server where I have installed Logstash 2.1.0
Here is my Logstash sample configuration file :-
input {
...
3
votes
2
answers
1k
views
Full data backup to Amazon S3
I have an Ubuntu server hosted on Digital Ocean that has outgrown it's existing backup solution.
The relevant parts of the stack I use are Node.js, MongoDB, and Elasticsearch.
So far backups have ...
3
votes
1
answer
3k
views
How to rename and join an elasticsearch server preserving data?
I tested elasticsearch and decided to make a cluster. To this I renamed the node and defined a cluster
cluster.name = mycluster
node.name = firstnode
When restarting the changes are taken into ...
3
votes
1
answer
3k
views
Why is this exclude_lines in filebeat excluding all logs?
I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-...
3
votes
1
answer
12k
views
elk stack error "unable to fetch mapping do you have indices matching the pattern"
i am trying to setup ELK stack with collectd on Ubuntu 16.04 LTS (so pretty much latest version of stack available)
kibana is behind nginx proxy (followed this guide https://www.digitalocean.com/...
3
votes
1
answer
3k
views
Using Logstash as shipper?
We are shipping logs from servers and using THE logstash on each server for shipping.
So we read logs from the glob "/root/Desktop/Logstash-Input/**/*_log".
input {
file{
...
3
votes
1
answer
5k
views
Logstash Server Log FIle Flooded With "Failed to flush outgoing items" Error Messages
I'm new to Logstash, trying to get it running following the tutorial in "The Logstash Book". On page 44, the guide suggests to tail the Logstash process' main log file called central.log. After ...