Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

learn more… | top users | synonyms

0
votes
1answer
15 views

Is it generally a bad idea to run MongoDB and ElasticSerach on the same nodes

Simple question: is it stupid to run mongodb and elasticsearch on the same nodes within a cluster of three if you take care of mongodb's memory hunger using cgroups and ignore the aspect of concurring ...
0
votes
0answers
14 views

Using database for storing text files and images

I need to build a image/static file cache which would host maybe few hundred up to thousand static files from 1kb to ~200kb. I'd need super fast reads (I'm using redis, but it's one threaded and I'm ...
0
votes
0answers
46 views

Install logstash forwarder

Any body please help me how to install logstash forwarder in 12.04 ubuntu version? I followed logstash book to install forwarder I got some ruby error these are the steps i followed $ sudo ...
0
votes
0answers
4 views

What the best practices to import Elasticsearch indices

I am running an elasticsearch cluster of 5 nodes in a production subnetwork, it's a "front" cluster with lot of search/read operations (DNS es-1.data.prod, es-2.data.prod and so on). In an other side, ...
6
votes
2answers
119 views

Sending Subversion logs to Logstash

My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own ...
2
votes
1answer
60 views

Elasticsearch performance tuning

In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge. Logstash conf ...
0
votes
0answers
33 views

How parse a multiple objects serialized in log with logstash and grok

I'm parsing some log files with grok, but I have the following trace: [es.org.some.Object[ attr1=53212 searchTags=[love, rock] ]] [es.org.some.Object[ attr1=54612 searchTags=[love, peace, rock] ...
6
votes
0answers
59 views

Is there disk based file system with dedicated memory caching for Linux? [closed]

We have two (or more sometimes) applications battling for the OS page cache. In particular ElasticSearch is very dependent on the OS page cache. To the point that cache misses can make or break ...
0
votes
1answer
85 views

Error while setting up grafana with graphite and nginx

While trying to setup grafana with graphite using nginx server. I am getting this error: No 'Access-Control-Allow-Origin' header is present on the requested resource. I tried changing configuration ...
1
vote
1answer
334 views

Error Could not contact Elasticsearch at http://localhost:9200. - kibana dashboard

'Error Could not contact Elasticsearch at http://localhost:9200. Please ensure that Elasticsearch is reachable from your system.' Currently I am following http://packetbeat.com/getstarted I ...
2
votes
1answer
45 views

Elasticsearch index much larger than the actual size of the logs it indexed?

I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like ...
0
votes
0answers
32 views

ElasticSearch snapshot backup sliding expiration - possible?

I'm planning on using the ElasticSearch S3 cloud plugin to create snapshots of our ES cluster. This all looks fairly straight forward but I'm wondering whether its possible integrate it into our ...
0
votes
0answers
49 views

Why doesn't logstash grab or index the files from the mapped drive?

I don't understand why logstash is so finicky with network resources. I shared a folder on another machine and then mapped it as Z: under Windows Explorer. I've verified the path and everything. I can ...
1
vote
1answer
69 views

How to rename and join an elasticsearch server preserving data?

I tested elasticsearch and decided to make a cluster. To this I renamed the node and defined a cluster cluster.name = mycluster node.name = firstnode When restarting the changes are taken into ...
0
votes
2answers
107 views

What is the purpose of elasticsearch logs? How to manage them?

As I understand indexes(or data) are being stored in /var/lib/elasticsearch by default, this folder contains nodes with 0 and 1 folders and overall size of these folders is 376M. The logs are ...
1
vote
2answers
201 views

Elasticsearch as memcache replacement

I am looking into replacing memcached with elasticsearch. Basically, I need to reduce my software stack and since I will be using elastic for searching, I was thinking of creating an index that will ...
0
votes
1answer
20 views

Access the JVM section of elasticsearch through the monitoring API

How I can access the jvm section of the node stats API ? I've read about it in the docs - monitoring individual nodes I'm doing a call to /_nodes/stats?pretty=true but don't have the jvm ...
-1
votes
1answer
42 views

Statistical information regarding scaling ElasticSearch [duplicate]

I have a set-up with several Logstash nodes sending input to ElasticSearch, and have a kibana server which allows me to visualize this. The current infrastructure is pretty simplistic and is on ...
0
votes
0answers
163 views

Kibana not finding local files e.g. require JS, CSS

I'm new to linux/ubuntu and the ELK stack and having an issue getting kibana to work. I've unpacked all three into this folder and started kibana and elastic search: cd /home/ubuntu ls ...
0
votes
0answers
788 views

logstash / kibana can't connect to instance

I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries. However.. today when I went to my logstash page, I see ...
0
votes
1answer
402 views

kibana: no indices error

None of the existing answers helped, so here a new question. Usecase: Redirecting syslog (or) monitoring static file. I have successfully installed logstash (1.4.2), elasticsearch(1.1.1) and ...
0
votes
1answer
292 views

Elasticsearch River JDBC - Avoiding 'SELECT *'

I'd like to syncronize some MySQL tables to an ElasticSearch index, you know, for search. But almost every examples I can find show a big bad fat "SELECT * FROM", it's OK for an example, but I'm in ...
0
votes
0answers
57 views

ElasticSearch cluster with remote nodes

Introduction Right now, I have three ElasticSearch servers (in Germany, US and Middle East(ME)). Each one has the same indexes but different data (they collect logs from local servers). Goal I want ...
0
votes
0answers
100 views

Reduce Elasticsearch log level

I have been trying to reduce the number of messages being logged by elastic search, however no matter what levels i put on each value (WARN, ERROR) i always see alot of DEBUG/WARN messages... I want ...
1
vote
1answer
2k views

Elasticsearch is using way too much disk space

I have a CentOS 6.5 server on which I installed Elasticsearch 1.3.2. My elasticsearch.yml configuration file is a minimal modification of the one shipping with elasticsearch as a default. Once ...
0
votes
0answers
194 views

Logstash/collectd buffer to files on network fault

I have read this: http://logstash.net/docs/1.4.2/life-of-an-event An example setup: collectd ( on local machine ) ----[via UDP]---> logstash (on local machine) ---[via elasticsearch output ...
0
votes
0answers
181 views

Kopf can not access elasticsearch

I have a setup with ElasticSearch, Logstash and Kibana working. With Kibana, I can access the data of the ElasticSearch from a remote host. I installed the latest kopf plugin with : ...
0
votes
1answer
6k views

Configure Nginx, Kibana, Elasticsearch

My setup Nginx + Kibana - same box different domain, sub-domain (nginx - example.com, kibana - kibana.example.com) Elasticsearch - 192.168.100.31 I have used the config defined below. Replaced ...
-2
votes
1answer
143 views

How do I know how the hacker gained access to my machine? [duplicate]

I own a dedicated machine and today I got an email from my host saying that my server will be blocked if I don't stop abusing the network. I am not a spammer or hacker. Someone was using my machine to ...
2
votes
1answer
119 views

ES fills up heap slowly over time and hangs at 14GB, while biggest index is 164MB?

I have an issue with Elasticsearch, at some times, it tries to run GC continously since this one is unable to free because heap size set to 14GB (min and max) is said to be completely allocated: ...
0
votes
0answers
32 views

ElasticSearch mappings for Apache URLs

So I'm feeding Apache access logs into my ElasticSearch cluster, but those darned developers keep crashing my cluster with their ad-hoc queries. Can anyone share what they're using to facilitate ...
0
votes
1answer
877 views

Installing Graylog2 with puppet

I have manged to install Graylog2 from scratch by downloading all the DEB files and making the configurations. Now I try to install Graylog2 with the help of puppet in an environment and had no luck ...
0
votes
1answer
291 views

How can I add basic auth to an apache proxy while using CORS headers?

Background I use grafana for displaying graphs of our server-metrics. Grafana is a JS-app which in our case is getting the data from graphite and storing search-queries in elasticsearch. All three ...
1
vote
0answers
332 views

ElasticSearch major slowdown upon big aggregations

My goal is to use ElasticSearch v1.3.2 for analyzing product cross-sales, so I need to filter for the receipts of interest (having an alcoholic product for example) and finding top selling products of ...
1
vote
0answers
366 views

JVM process resident set size “equals” max heap size, not current heap size

After a few reading about jvm memory (here, here, here, others I forgot...), I am expecting the resident set size of my java process to be roughly equal to the current heap space capacity. That's not ...
0
votes
1answer
480 views

Could not access pid file for elasticsearch

I am not a die-hard linux engineer, but I am having some problems with starting Elastic Search. Some stats: Server: Ubuntu Server 11.04 Elastic Search: 1.2.3 (installed with appitude) When I start ...
0
votes
1answer
86 views

Monitoring solution that doesn't duplicate source data?

I need to analyze data stored in multiple databases. Each of them contains a few TB of logs and sensor data. If I use Splunk or ElasticSearch/Kibana, I see 2 solutions: Batch import everything ...
1
vote
1answer
852 views

Graylog2 server cannot connect to Elasticsearch

I try to follow Digital Ocean manual except using last versions of components. My elasticsearch.yml: cluster.name: graylog2 network.bind_host: localhost network.publish_host: localhost ...
1
vote
1answer
605 views

Elasticsearch Cluster State immediately goes Yellow after first index created

I run a number of standalone Logstash servers to allow review of log files from web application servers. One of these recently reported a Yellow cluster state due to unassigned shards. This is a ...
0
votes
1answer
628 views

How do I make a graceful shutdown (with zero replicas)?

I am running a three node (elasticsearch) cluster currently with index.number_of_replicas: 0. I'd like to bring one of the node down for maintenance. Do I have any options to do this without ...
0
votes
0answers
148 views

ElasticSearch upgrade from 0.90.5 to 1.2.2

we're planning an Elasticsearch cluster upgrade from 0.90.5 to 1.2.2 . As this will be my first "full cluster restart" upgrade just wanted to get some information that a procedure I've put in place is ...
2
votes
3answers
888 views

sudden peak in cpu usage

We are running a 4 node/machine elastic search cluster on 12 core, 96gb RAM, 4 spinning disk machines. under normal operation most cpu usage is user and around 5-10%. Every few days, one of the ...
0
votes
1answer
159 views

Is it possible to do autoscaling for elasticsearch in AWS at peak load?

I'm new to ElasticSearch but I'm already using ElasticSearch on my production site. I installed ElasticSearch in one EC2 instance. After using ElasticSearch for several month, now currently the CPU ...
0
votes
1answer
808 views

Numerical range in Kibana 3 not working

My tomcat access logs have last field as "%D - Time taken to process the request, in millis". I am trying to filter the logs which took more than some number of milliseconds, but it is showing all the ...
0
votes
0answers
109 views

indexing a ~3gb database into elasticsearch: how?

I have a new database of 2 million records (around 3gb when dumped json) produced every several days. I want to quickly put it into ElasticSearch. That's what I do right now: Create a new index and ...
3
votes
1answer
1k views

Using Logstash as shipper?

We are shipping logs from servers and using THE logstash on each server for shipping. So we read logs from the glob "/root/Desktop/Logstash-Input/**/*_log". input { file{ ...
0
votes
1answer
2k views

Problems configuring logstash for email output

I'm trying to configure logstash to send email alerts and log output in elasticsearch / kibana. I have the logs successfully syncing via rsyslog, but I get the following error when I run ...
1
vote
1answer
675 views

Iptables config breaks Java + Elastic Search communication

I am trying to set up a firewall for a server hosting a java app and ES. Both are on the same server and communicate to each other. The problem I am having is that my firewall configuration prevents ...
1
vote
1answer
2k views

Unable to configure Nginx as a reverse proxy for elasticsearch on another server

I'm trying to use Kibana on a server A to access elasticsearch on a server B. elasticsearch is protected with a basic auth authentication on server B. Kibana is also protected with basic ...
0
votes
1answer
359 views

iptables to allow input and output traffic to and from web server only

I have an Elastic Search server which seems to have been exploited (it's being used for a DDoS attack having had NO firewall for about a month). As a temporary measure while I create a new one I was ...