Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

learn more… | top users | synonyms

2
votes
0answers
19 views

Benchmarking for DISK I/O Writes Per Second - ElasticSearch and MySQL

During working on our project we needed to decide should we use "ElasticSearch" or "MySQL" to record the audit/trail logs of the system. Searching speed is not an issue here, we needed to see the disk ...
0
votes
0answers
28 views

Kibana 3 on nginx with ssl

I installed Kibana 3 on nginx with the nginx-auth-ldap module. And with the example nginx.conf provided by elasticsearch and added ssl support. But for some reason, when accessing kibana with https I ...
0
votes
0answers
26 views

elasticsearch not enabled when installed on additional device

Here's the deal : I'm working with AWS EC2 instances (Centos 7). Elasticsearch is installed from the sources in the folder /app. The elasticsearch service is managed by init.d. The problem is that if ...
0
votes
2answers
45 views

elasticsearch-river-jdbc inserting duplicate record in mysql db

Sorry I am newbie to Elasticsearch and I am using elasticsearch-river-jdbc to connect to myql database,everything is working fine except for the fact that everytime it run as per schedule its ...
0
votes
1answer
43 views

Is it generally a bad idea to run MongoDB and ElasticSerach on the same nodes

Simple question: is it stupid to run mongodb and elasticsearch on the same nodes within a cluster of three if you take care of mongodb's memory hunger using cgroups and ignore the aspect of concurring ...
2
votes
0answers
33 views

Using database for storing text files and images [closed]

I need to build a image/static file cache which would host maybe few hundred up to thousand static files from 1kb to ~200kb. I'd need super fast reads (I'm using redis, but it's one threaded and I'm ...
0
votes
0answers
114 views

Install logstash forwarder

Any body please help me how to install logstash forwarder in 12.04 ubuntu version? I followed logstash book to install forwarder I got some ruby error these are the steps i followed $ sudo ...
0
votes
0answers
22 views

What the best practices to import Elasticsearch indices

I am running an elasticsearch cluster of 5 nodes in a production subnetwork, it's a "front" cluster with lot of search/read operations (DNS es-1.data.prod, es-2.data.prod and so on). In an other side, ...
6
votes
2answers
149 views

Sending Subversion logs to Logstash

My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own ...
2
votes
1answer
111 views

Elasticsearch performance tuning

In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge. Logstash conf ...
0
votes
0answers
43 views

How parse a multiple objects serialized in log with logstash and grok

I'm parsing some log files with grok, but I have the following trace: [es.org.some.Object[ attr1=53212 searchTags=[love, rock] ]] [es.org.some.Object[ attr1=54612 searchTags=[love, peace, rock] ...
6
votes
0answers
67 views

Is there disk based file system with dedicated memory caching for Linux? [closed]

We have two (or more sometimes) applications battling for the OS page cache. In particular ElasticSearch is very dependent on the OS page cache. To the point that cache misses can make or break ...
0
votes
1answer
250 views

Error while setting up grafana with graphite and nginx

While trying to setup grafana with graphite using nginx server. I am getting this error: No 'Access-Control-Allow-Origin' header is present on the requested resource. I tried changing configuration ...
1
vote
1answer
709 views

Error Could not contact Elasticsearch at http://localhost:9200. - kibana dashboard

'Error Could not contact Elasticsearch at http://localhost:9200. Please ensure that Elasticsearch is reachable from your system.' Currently I am following http://packetbeat.com/getstarted I ...
2
votes
1answer
64 views

Elasticsearch index much larger than the actual size of the logs it indexed?

I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like ...
0
votes
0answers
41 views

ElasticSearch snapshot backup sliding expiration - possible?

I'm planning on using the ElasticSearch S3 cloud plugin to create snapshots of our ES cluster. This all looks fairly straight forward but I'm wondering whether its possible integrate it into our ...
0
votes
0answers
80 views

Why doesn't logstash grab or index the files from the mapped drive?

I don't understand why logstash is so finicky with network resources. I shared a folder on another machine and then mapped it as Z: under Windows Explorer. I've verified the path and everything. I can ...
1
vote
1answer
151 views

How to rename and join an elasticsearch server preserving data?

I tested elasticsearch and decided to make a cluster. To this I renamed the node and defined a cluster cluster.name = mycluster node.name = firstnode When restarting the changes are taken into ...
0
votes
2answers
168 views

What is the purpose of elasticsearch logs? How to manage them?

As I understand indexes(or data) are being stored in /var/lib/elasticsearch by default, this folder contains nodes with 0 and 1 folders and overall size of these folders is 376M. The logs are ...
1
vote
2answers
227 views

Elasticsearch as memcache replacement

I am looking into replacing memcached with elasticsearch. Basically, I need to reduce my software stack and since I will be using elastic for searching, I was thinking of creating an index that will ...
0
votes
1answer
26 views

Access the JVM section of elasticsearch through the monitoring API

How I can access the jvm section of the node stats API ? I've read about it in the docs - monitoring individual nodes I'm doing a call to /_nodes/stats?pretty=true but don't have the jvm ...
-1
votes
1answer
52 views

Statistical information regarding scaling ElasticSearch [duplicate]

I have a set-up with several Logstash nodes sending input to ElasticSearch, and have a kibana server which allows me to visualize this. The current infrastructure is pretty simplistic and is on ...
0
votes
0answers
196 views

Kibana not finding local files e.g. require JS, CSS

I'm new to linux/ubuntu and the ELK stack and having an issue getting kibana to work. I've unpacked all three into this folder and started kibana and elastic search: cd /home/ubuntu ls ...
1
vote
0answers
959 views

logstash / kibana can't connect to instance

I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries. However.. today when I went to my logstash page, I see ...
0
votes
1answer
601 views

kibana: no indices error

None of the existing answers helped, so here a new question. Usecase: Redirecting syslog (or) monitoring static file. I have successfully installed logstash (1.4.2), elasticsearch(1.1.1) and ...
0
votes
1answer
366 views

Elasticsearch River JDBC - Avoiding 'SELECT *'

I'd like to syncronize some MySQL tables to an ElasticSearch index, you know, for search. But almost every examples I can find show a big bad fat "SELECT * FROM", it's OK for an example, but I'm in ...
0
votes
0answers
64 views

ElasticSearch cluster with remote nodes

Introduction Right now, I have three ElasticSearch servers (in Germany, US and Middle East(ME)). Each one has the same indexes but different data (they collect logs from local servers). Goal I want ...
0
votes
0answers
113 views

Reduce Elasticsearch log level

I have been trying to reduce the number of messages being logged by elastic search, however no matter what levels i put on each value (WARN, ERROR) i always see alot of DEBUG/WARN messages... I want ...
1
vote
1answer
3k views

Elasticsearch is using way too much disk space

I have a CentOS 6.5 server on which I installed Elasticsearch 1.3.2. My elasticsearch.yml configuration file is a minimal modification of the one shipping with elasticsearch as a default. Once ...
0
votes
0answers
228 views

Logstash/collectd buffer to files on network fault

I have read this: http://logstash.net/docs/1.4.2/life-of-an-event An example setup: collectd ( on local machine ) ----[via UDP]---> logstash (on local machine) ---[via elasticsearch output ...
0
votes
0answers
212 views

Kopf can not access elasticsearch

I have a setup with ElasticSearch, Logstash and Kibana working. With Kibana, I can access the data of the ElasticSearch from a remote host. I installed the latest kopf plugin with : ...
0
votes
1answer
6k views

Configure Nginx, Kibana, Elasticsearch

My setup Nginx + Kibana - same box different domain, sub-domain (nginx - example.com, kibana - kibana.example.com) Elasticsearch - 192.168.100.31 I have used the config defined below. Replaced ...
-2
votes
1answer
151 views

How do I know how the hacker gained access to my machine? [duplicate]

I own a dedicated machine and today I got an email from my host saying that my server will be blocked if I don't stop abusing the network. I am not a spammer or hacker. Someone was using my machine to ...
2
votes
1answer
148 views

ES fills up heap slowly over time and hangs at 14GB, while biggest index is 164MB?

I have an issue with Elasticsearch, at some times, it tries to run GC continously since this one is unable to free because heap size set to 14GB (min and max) is said to be completely allocated: ...
0
votes
0answers
33 views

ElasticSearch mappings for Apache URLs

So I'm feeding Apache access logs into my ElasticSearch cluster, but those darned developers keep crashing my cluster with their ad-hoc queries. Can anyone share what they're using to facilitate ...
0
votes
1answer
961 views

Installing Graylog2 with puppet

I have manged to install Graylog2 from scratch by downloading all the DEB files and making the configurations. Now I try to install Graylog2 with the help of puppet in an environment and had no luck ...
0
votes
1answer
349 views

How can I add basic auth to an apache proxy while using CORS headers?

Background I use grafana for displaying graphs of our server-metrics. Grafana is a JS-app which in our case is getting the data from graphite and storing search-queries in elasticsearch. All three ...
1
vote
0answers
358 views

ElasticSearch major slowdown upon big aggregations

My goal is to use ElasticSearch v1.3.2 for analyzing product cross-sales, so I need to filter for the receipts of interest (having an alcoholic product for example) and finding top selling products of ...
1
vote
0answers
409 views

JVM process resident set size “equals” max heap size, not current heap size

After a few reading about jvm memory (here, here, here, others I forgot...), I am expecting the resident set size of my java process to be roughly equal to the current heap space capacity. That's not ...
0
votes
1answer
550 views

Could not access pid file for elasticsearch

I am not a die-hard linux engineer, but I am having some problems with starting Elastic Search. Some stats: Server: Ubuntu Server 11.04 Elastic Search: 1.2.3 (installed with appitude) When I start ...
0
votes
1answer
93 views

Monitoring solution that doesn't duplicate source data?

I need to analyze data stored in multiple databases. Each of them contains a few TB of logs and sensor data. If I use Splunk or ElasticSearch/Kibana, I see 2 solutions: Batch import everything ...
1
vote
1answer
1k views

Graylog2 server cannot connect to Elasticsearch

I try to follow Digital Ocean manual except using last versions of components. My elasticsearch.yml: cluster.name: graylog2 network.bind_host: localhost network.publish_host: localhost ...
1
vote
1answer
713 views

Elasticsearch Cluster State immediately goes Yellow after first index created

I run a number of standalone Logstash servers to allow review of log files from web application servers. One of these recently reported a Yellow cluster state due to unassigned shards. This is a ...
0
votes
1answer
764 views

How do I make a graceful shutdown (with zero replicas)?

I am running a three node (elasticsearch) cluster currently with index.number_of_replicas: 0. I'd like to bring one of the node down for maintenance. Do I have any options to do this without ...
0
votes
0answers
156 views

ElasticSearch upgrade from 0.90.5 to 1.2.2

we're planning an Elasticsearch cluster upgrade from 0.90.5 to 1.2.2 . As this will be my first "full cluster restart" upgrade just wanted to get some information that a procedure I've put in place is ...
2
votes
3answers
1k views

sudden peak in cpu usage

We are running a 4 node/machine elastic search cluster on 12 core, 96gb RAM, 4 spinning disk machines. under normal operation most cpu usage is user and around 5-10%. Every few days, one of the ...
0
votes
1answer
199 views

Is it possible to do autoscaling for elasticsearch in AWS at peak load?

I'm new to ElasticSearch but I'm already using ElasticSearch on my production site. I installed ElasticSearch in one EC2 instance. After using ElasticSearch for several month, now currently the CPU ...
0
votes
1answer
989 views

Numerical range in Kibana 3 not working

My tomcat access logs have last field as "%D - Time taken to process the request, in millis". I am trying to filter the logs which took more than some number of milliseconds, but it is showing all the ...
0
votes
0answers
121 views

indexing a ~3gb database into elasticsearch: how?

I have a new database of 2 million records (around 3gb when dumped json) produced every several days. I want to quickly put it into ElasticSearch. That's what I do right now: Create a new index and ...
3
votes
1answer
1k views

Using Logstash as shipper?

We are shipping logs from servers and using THE logstash on each server for shipping. So we read logs from the glob "/root/Desktop/Logstash-Input/**/*_log". input { file{ ...