Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

learn more… | top users | synonyms

0
votes
0answers
16 views

Numerical range in Kibana 3 not working

My tomcat access logs have last field as "%D - Time taken to process the request, in millis". I am trying to filter the logs which took more than some number of milliseconds, but it is showing all the ...
0
votes
0answers
24 views

indexing a ~3gb database into elasticsearch: how?

I have a new database of 2 million records (around 3gb when dumped json) produced every several days. I want to quickly put it into ElasticSearch. That's what I do right now: Create a new index and ...
1
vote
1answer
35 views

Using Logstash as shipper?

We are shipping logs from servers and using THE logstash on each server for shipping. So we read logs from the glob "/root/Desktop/Logstash-Input/**/*_log". input { file{ ...
0
votes
1answer
187 views

Problems configuring logstash for email output

I'm trying to configure logstash to send email alerts and log output in elasticsearch / kibana. I have the logs successfully syncing via rsyslog, but I get the following error when I run ...
0
votes
1answer
130 views

Iptables config breaks Java + Elastic Search communication

I am trying to set up a firewall for a server hosting a java app and ES. Both are on the same server and communicate to each other. The problem I am having is that my firewall configuration prevents ...
0
votes
1answer
421 views

Unable to configure Nginx as a reverse proxy for elasticsearch on another server

I'm trying to use Kibana on a server A to access elasticsearch on a server B. elasticsearch is protected with a basic auth authentication on server B. Kibana is also protected with basic ...
0
votes
1answer
174 views

iptables to allow input and output traffic to and from web server only

I have an Elastic Search server which seems to have been exploited (it's being used for a DDoS attack having had NO firewall for about a month). As a temporary measure while I create a new one I was ...
0
votes
2answers
55 views

What are records in 'netstat' command are when using logstash?

I use logstash with embedded elasticsearch. My logstash config looks like the following: output { tdout { debug => true debug_format => "json"} elasticsearch { embedded => true ...
1
vote
2answers
92 views

PHP - does curl run asynchroniously on my system? [closed]

I'm building a web app. I have a database of books indexed in ElasticSearch and REST API written in PHP. In the app, there's a search box, where I type a name of the book and JS script calls the ...
2
votes
0answers
273 views

Unassigned shards in elasticsearch

I have five nodes : Two Master Nodes, One Balancer Node, One Workhorse Node, and One Coordinator Node. I am shipping events from logstash, redis, to elasticsearch. At the moment, my cluster is RED. ...
0
votes
1answer
814 views

elasticsearch with kibana displays empty dashboard

I have the following is showing up when I pull up my kibana dashboard: http://192.168.10.25/#/dashboard {{dashboard.current.title}} When I tail my logs I see the following 2014/05/14 13:31:45 ...
0
votes
0answers
57 views

rsyslog - How to flush queue?

I'm using rsyslog to get logs from haproxy and put them into logstash for elasticsearch/kibana. Everything was working ok, but i've found something strange in rsyslog. I've found that i've got ...
1
vote
1answer
208 views

How best to monitor logstash?

I've seen this question on the mailing list a few times but haven't had a satisfactory answer. How best to monitor that the pipeline isn't stuck? Clients -> logstash -> elasticsearch. Logstash and ...
0
votes
1answer
132 views

Can I run elasticsearch on a single server?

I use elasticsearch as part of a Logstash stack, in which all of the components of the stack are installed on the same server. The purpose of this is to expose application logs to developers for ...
0
votes
0answers
32 views

What should I use to monitor elasticsearch health?

I am planning on using elasticsearch in production for the first time. What statistics is it recommended that I monitor? I am currently monitoring: /_nodes/stats for jvm.mem.heap_used_percent over ...
1
vote
0answers
74 views

Elasticsearch on EC2, webserver on Heroku // EC2 ports only accessible from Heroku

I have an instance of Elasticsearch facing the internet on an EC2. On the other hand I have my webservice as a Heroku app. I want to secure the Elasticsearch instance and allow inbound traffic only ...
1
vote
1answer
1k views

Nothing appearing in kibana dashboard

I have installed Logstash ElasticSearch Kibana on an EC2 instance. I can visit http://example.com:9200 Here I get { "status" : 200, "name" : "Aleta Ogord", "version" : { "number" : ...
0
votes
1answer
259 views

Connection hangs & never connects between ec2 instances over port 9200; fast from outside AWS

I am trying to connect from a web server EC2 instance to an ElasticSearch server ec2 instance. The connection is slow to nonexistent when connecting from the EC2, but very fast when connecting from a ...
2
votes
1answer
679 views

ElasticSearch Server Randomly Stops Working

I have 2 ES servers that are being fed by 1 logstash server and viewing the logs in Kibana. This is a POC to work out any issues before going into production. The system has ran for ~1 month and every ...
-1
votes
1answer
172 views

Elasticsearch throws Could not reserve enough space for object heap

It seems that Elasticsearch stops working this morning. When i launch : sudo service elasticsearch start, it says : Starting ElasticSearch Server Done But when i check the status, it gives me : ...
0
votes
1answer
147 views

Elasticsearch stops working suddenly

I have deployed Elasticsearch on a development ubunto VPS. We use that VPS only to host Api for mobile apps. Everything worked good till today, elasticsearch stops working. I tried to restart it, it ...
0
votes
1answer
288 views

ElasticSearch Multiple Indexes and Routing

Currently I have the following setup. Syslog --> Logstash --> ElasticSearch --> Kibana Logstash is creating a daily index ...
0
votes
0answers
208 views

Output Logstach to Elasticsearch cluster

I have multiple EC2 instances running Logstash and Elasticsearch. I'd like to output everything into ES cluster. output { stdout { debug => true debug_format => "json" } ...
0
votes
1answer
293 views

Multiple instances of Logstash + Elasticsearch on AWS

I have setup a Cloudformation template to start an instance where Logstash and Elasticsearch (not embedded) are installed. I have standard tcp input configured on Logstash and output it to an ...
0
votes
0answers
117 views

graylog elasticsearch install cannot change open file limit on reboot

I have a problem with our graylog 2 install. The elasticsearch process keeps on running out of file handles, this is in the //opt/elasticsearch/logs/graylog2.log log file. [2014-01-09 ...
3
votes
1answer
1k views

Logstash Server Log FIle Flooded With “Failed to flush outgoing items” Error Messages

I'm new to Logstash, trying to get it running following the tutorial in "The Logstash Book". On page 44, the guide suggests to tail the Logstash process' main log file called central.log. After ...
3
votes
1answer
112 views

Securing elasticsearch

There appears to be a serious lack of information on this despite the fact that as installed elasticsearch is extremely exploitable. My main fear in using it is that as a non-expert I have no idea ...
2
votes
1answer
139 views

Logstash Groking Syslog Events

I have a logstash server sending events to an elasticsearch server that is displaying the results in Kibana. Everything is working pretty good, other than Kibana is showing 2 names in the host field. ...
1
vote
1answer
442 views

Elasticsearch standalone mode with logstash reject data after five days

I use Logstash and Elasticsearch for our squid log store and analyze.The size of the logs increase at the speed of 40Gb per day in our freeBSD ZFS storage system. the Elasticsearch gets into failure ...
2
votes
2answers
363 views

expose docker container port to eth0

First: could someone with 300 rep make a tag for "docker" please? TYVM I'm running a docker container for elasticsearch with the following command: docker run -d -i -t -p 9200:9200 -p 9300:9300 ...
0
votes
1answer
127 views

Launch elasticsearch dockerfile using my own elasticsearch.yml

I am launching elasticsearch via a dockerfile found here: https://index.docker.io/u/ehazlett/elasticsearch/ It works great. I need to define my own hosts as my environment does not support ...
1
vote
1answer
616 views

Large virtual memory size of ElasticSearch JVM

I am running a JVM to support ElasticSearch. I am still working on sizing and tuning, so I left the JVM's max heap size at ElasticSearch's default of 1GB. After putting data in the database, I find ...
2
votes
2answers
542 views

Foward slash in kibana 3 query

I'm trying to add a query that will match a request that ends with a slash, like this one: n.n.n.n - - [16/Oct/2013:16:40:41 +0100] "GET / HTTP/1.1" 200 25058 "-" "Mozilla/5.0 (iPad; CPU OS 7_0_2 ...
0
votes
0answers
213 views

Getting strange error from elasticsearch backed of LogStash

When i try to run any query from logstash like "httpd" search from url: http://mydomain.domain.com:9292/search , it gives me following error: "No results for query 'httpd'" And in the log ...
3
votes
1answer
2k views

How well does elasticsearch compress data?

I am looking to scope the servers required for an elasticsearch proof-of-concept. Ultimately, my question is this: Given 1GB of json text indexed by elasticsearch, how much can I expect to be ...
3
votes
1answer
495 views

Adding TTL to elasticsearch with logstash

I can't figure out how to apply this documentation http://www.elasticsearch.org/guide/reference/mapping/ttl-field/ to logstash. Specifically, i don't know which keys to use (instead of "tweet" etc). ...
2
votes
0answers
151 views

How do I enable digest auth for Jetty fronting Elastic Search

We've gotten Jetty running the HTTP front end of Elastic Search with BASIC authentication. We would like to use DIGEST authentication instead, but the Jetty documentation is hard to reconcile with the ...
6
votes
2answers
1k views

Secure logstash and elasticsearch

I'm considering running logstash on my prod server (simple install. http://logstash.net/docs/1.1.13/tutorials/getting-started-simple) and set kibana to access logs. My concern is: how to secure my ...
0
votes
2answers
208 views

Nginx stops serving passenger app (503)

I'm deploying a rails app through passenger and nginx (utilizing an elastic-search server which is running on the same machine) on a Ubuntu system. This works perfectly for about twelve hours. Then, ...
1
vote
0answers
106 views

Elasticsearch and couchdb river indexing slows down after a few hours

Trying to import ~400m docs in to Elasticsearch from couchdb using the couchdb river plugin. Everything starts out great with indexing time around 5k/s but after a few hours come back and find its ...
-1
votes
1answer
214 views

Which apt-get package of java do I install if I want to use elasticsearch?

sudo apt-get install XXX? What java package should I install?
1
vote
1answer
726 views

How much of hardware resource is required for Graylog2? [closed]

We've mounted a graylog2 dedicated server (with a rails unicorn, mongodb and eleasticsearch) on a virtual machine with 2gigs of ram a couple days ago. RAM Consumption just keeps climbing, I am ...
10
votes
1answer
6k views

Scaling Logstash (with redis/elasticsearch)

Over a cluster of over 12 centos 5.8 servers, I deployed logstash using the native logstash shipper, which sends /var/log/*/*.log back to a central logstash server. We tried using rsyslogd as the ...
3
votes
1answer
921 views

Migrating to AWS Cloud with auto-scaling - where to put Redis and ElasticSearch?

I've been trying to research this topic but haven't found anywhere that recommends where to install services such as Redis and ElasticSearch when migrating to a cloud framework. I'm currently running ...
0
votes
2answers
1k views

Elasticsearch won't start anymore

I restarted my elasticsearch instance 5 days ago and I haven't manage to start it since then. I get no output in the log file /var/log/elasticsearch/ nor does the elasticsearch binary print any ...
1
vote
2answers
440 views

Collecting logs from multiple servers in a fault-tolerant way [closed]

the main requirement I have is to be able to view and do simple searches on logs combined from multiple machines. However, I would like the solution to have minimal impact on the rest of the (core) ...
1
vote
1answer
499 views

elasticsearch @ debian6 started by /etc/init.d max_file_descriptors

I have installed elasticsearch @ debian6 by this tutorial. I have set up my system max open files directives so i have these values: # su # cat /proc/sys/fs/file-max 70000 # ulimit -Hn 64000 # ...
1
vote
1answer
195 views

Is it possible to deploy elasticsearch in Webfaction?

I'd like to deploy elasticsearch (ES) at Webfaction server. I have 256MB of memory and running a Django app that takes about 100MB. I need to index about 10,000 records. Is it possible to run ES in a ...
2
votes
1answer
198 views

Disk / system configuration for log collection / syslog server

I am looking into building a syslog / logging infrastructure and am pondering about some architecture best practices. Essentially, I see that a syslog system needs to support two conflicting ...
-1
votes
1answer
150 views

why did i need to stop java software? [closed]

i am on Debian Squeeze, 4G of Ram, 1G of Swap, CPU i3 3.07GHz. I used to use the following java softwares: jenkins elasticsearch Problem was their process was too greedy about CPU, in the top ...