Questions tagged [elasticsearch]
Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.
379
questions
7
votes
4
answers
26k
views
Error Could not contact Elasticsearch at http://localhost:9200. - kibana dashboard
'Error Could not contact Elasticsearch at http://localhost:9200. Please ensure that Elasticsearch is reachable from your system.'
Currently I am following http://packetbeat.com/getstarted
I ...
- 171
1
vote
2
answers
986
views
Elasticsearch index much larger than the actual size of the logs it indexed?
I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like 2....
4
votes
1
answer
895
views
ElasticSearch snapshot backup sliding expiration - possible?
I'm planning on using the ElasticSearch S3 cloud plugin to create snapshots of our ES cluster. This all looks fairly straight forward but I'm wondering whether its possible integrate it into our ...
- 141
0
votes
1
answer
2k
views
Why doesn't logstash grab or index the files from the mapped drive?
I don't understand why logstash is so finicky with network resources. I shared a folder on another machine and then mapped it as Z: under Windows Explorer. I've verified the path and everything. I can ...
3
votes
1
answer
3k
views
How to rename and join an elasticsearch server preserving data?
I tested elasticsearch and decided to make a cluster. To this I renamed the node and defined a cluster
cluster.name = mycluster
node.name = firstnode
When restarting the changes are taken into ...
- 3,657
5
votes
2
answers
7k
views
What is the purpose of elasticsearch logs? How to manage them?
As I understand indexes(or data) are being stored in
/var/lib/elasticsearch
by default, this folder contains nodes with 0 and 1 folders and overall size of these folders is 376M.
The logs are ...
- 225
1
vote
2
answers
3k
views
Elasticsearch as memcache replacement
I am looking into replacing memcached with elasticsearch. Basically, I need to reduce my software stack and since I will be using elastic for searching, I was thinking of creating an index that will ...
- 167
0
votes
1
answer
163
views
Access the JVM section of elasticsearch through the monitoring API
How I can access the jvm section of the node stats API ?
I've read about it in the docs - monitoring individual nodes
I'm doing a call to
/_nodes/stats?pretty=true
but don't have the jvm section ...
- 307
0
votes
1
answer
244
views
Statistical information regarding scaling ElasticSearch [duplicate]
I have a set-up with several Logstash nodes sending input to ElasticSearch, and have a kibana server which allows me to visualize this.
The current infrastructure is pretty simplistic and is on ...
- 121
1
vote
0
answers
2k
views
logstash / kibana can't connect to instance
I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries.
However.. today when I went to my logstash page, I see ...
- 307
3
votes
3
answers
3k
views
kibana: no indices error
None of the existing answers helped, so here a new question.
Usecase: Redirecting syslog (or) monitoring static file.
I have successfully installed logstash (1.4.2), elasticsearch(1.1.1) and kibana(...
- 31
0
votes
1
answer
971
views
Elasticsearch River JDBC - Avoiding 'SELECT *'
I'd like to syncronize some MySQL tables to an ElasticSearch index, you know, for search.
But almost every examples I can find show a big bad fat "SELECT * FROM", it's OK for an example, but I'm in ...
- 251
13
votes
1
answer
37k
views
Elasticsearch is using way too much disk space
I have a CentOS 6.5 server on which I installed Elasticsearch 1.3.2.
My elasticsearch.yml configuration file is a minimal modification of the one shipping with elasticsearch as a default. Once ...
- 500
0
votes
1
answer
12k
views
Configure Nginx, Kibana, Elasticsearch
My setup
Nginx + Kibana - same box different domain, sub-domain (nginx - example.com, kibana - kibana.example.com)
Elasticsearch - 192.168.100.31
I have used the config defined below. Replaced 127....
- 133
-1
votes
1
answer
795
views
How do I know how the hacker gained access to my machine? [duplicate]
I own a dedicated machine and today I got an email from my host saying that my server will be blocked if I don't stop abusing the network. I am not a spammer or hacker. Someone was using my machine to ...
- 109
2
votes
1
answer
1k
views
ES fills up heap slowly over time and hangs at 14GB, while biggest index is 164MB?
I have an issue with Elasticsearch, at some times, it tries to run GC continously since this one is unable to free because heap size set to 14GB (min and max) is said to be completely allocated:
(...)...
- 858
1
vote
1
answer
2k
views
Installing Graylog2 with puppet
I have manged to install Graylog2 from scratch by downloading all the DEB files and making the configurations. Now I try to install Graylog2 with the help of puppet in an environment and had no luck ...
- 123
1
vote
1
answer
3k
views
How can I add basic auth to an apache proxy while using CORS headers?
Background
I use grafana for displaying graphs of our server-metrics. Grafana is a JS-app which in our case is getting the data from graphite and storing search-queries in elasticsearch. All three ...
- 182
2
votes
0
answers
819
views
ElasticSearch major slowdown upon big aggregations
My goal is to use ElasticSearch v1.3.2 for analyzing product cross-sales, so I need to filter for the receipts of interest (having an alcoholic product for example) and finding top selling products of ...
- 161
1
vote
0
answers
3k
views
JVM process resident set size "equals" max heap size, not current heap size
After a few reading about jvm memory (here, here, here, others I forgot...), I am expecting the resident set size of my java process to be roughly equal to the current heap space capacity. That's not ...
- 111
1
vote
1
answer
2k
views
Could not access pid file for elasticsearch
I am not a die-hard linux engineer, but I am having some problems with starting Elastic Search. Some stats:
Server: Ubuntu Server 11.04
Elastic Search: 1.2.3 (installed with appitude)
When I start ...
- 141
0
votes
1
answer
273
views
Monitoring solution that doesn't duplicate source data?
I need to analyze data stored in multiple databases. Each of them contains a few TB of logs and sensor data.
If I use Splunk or ElasticSearch/Kibana, I see 2 solutions:
Batch import everything
Write ...
- 313
1
vote
1
answer
5k
views
Graylog2 server cannot connect to Elasticsearch
I try to follow Digital Ocean manual except using last versions of components.
My elasticsearch.yml:
cluster.name: graylog2
network.bind_host: localhost
network.publish_host: localhost
script....
- 43
1
vote
1
answer
3k
views
Elasticsearch Cluster State immediately goes Yellow after first index created
I run a number of standalone Logstash servers to allow review of log files from web application servers.
One of these recently reported a Yellow cluster state due to unassigned shards. This is a ...
- 3,529
1
vote
1
answer
5k
views
How do I make a graceful shutdown (with zero replicas)?
I am running a three node (elasticsearch) cluster currently with index.number_of_replicas: 0. I'd like to bring one of the node down for maintenance. Do I have any options to do this without ...
- 1,405
2
votes
3
answers
3k
views
sudden peak in cpu usage
We are running a 4 node/machine elastic search cluster on 12 core, 96gb RAM, 4 spinning disk machines. under normal operation most cpu usage is user and around 5-10%. Every few days, one of the ...
- 71
0
votes
1
answer
3k
views
Is it possible to do autoscaling for elasticsearch in AWS at peak load?
I'm new to ElasticSearch but I'm already using ElasticSearch on my production site.
I installed ElasticSearch in one EC2 instance. After using ElasticSearch for several month, now currently the CPU ...
- 121
0
votes
1
answer
4k
views
Numerical range in Kibana 3 not working
My tomcat access logs have last field as "%D - Time taken to process the request, in millis".
I am trying to filter the logs which took more than some number of milliseconds, but it is showing all the ...
- 691
0
votes
0
answers
356
views
indexing a ~3gb database into elasticsearch: how?
I have a new database of 2 million records (around 3gb when dumped json) produced every several days. I want to quickly put it into ElasticSearch.
That's what I do right now:
Create a new index and ...
- 203
3
votes
1
answer
3k
views
Using Logstash as shipper?
We are shipping logs from servers and using THE logstash on each server for shipping.
So we read logs from the glob "/root/Desktop/Logstash-Input/**/*_log".
input {
file{
...
- 149
1
vote
2
answers
6k
views
Problems configuring logstash for email output
I'm trying to configure logstash to send email alerts and log output in elasticsearch / kibana.
I have the logs successfully syncing via rsyslog, but I get the following error when I run
/opt/...
- 153
1
vote
1
answer
2k
views
Iptables config breaks Java + Elastic Search communication
I am trying to set up a firewall for a server hosting a java app and ES. Both are on the same server and communicate to each other. The problem I am having is that my firewall configuration prevents ...
- 133
1
vote
1
answer
5k
views
Unable to configure Nginx as a reverse proxy for elasticsearch on another server
I'm trying to use Kibana on a server A to access elasticsearch on a server B.
elasticsearch is protected with a basic auth authentication on server B.
Kibana is also protected with basic ...
- 113
2
votes
1
answer
2k
views
iptables to allow input and output traffic to and from web server only
I have an Elastic Search server which seems to have been exploited (it's being used for a DDoS attack having had NO firewall for about a month).
As a temporary measure while I create a new one I was ...
- 123
1
vote
2
answers
3k
views
What are records in 'netstat' command are when using logstash?
I use logstash with embedded elasticsearch. My logstash config looks like the following:
output {
tdout { debug => true debug_format => "json"}
elasticsearch {
embedded => true
...
- 213
2
votes
2
answers
6k
views
PHP - does curl run asynchroniously on my system? [closed]
I'm building a web app. I have a database of books indexed in ElasticSearch and REST API written in PHP.
In the app, there's a search box, where I type a name of the book and JS script calls the ...
- 175
2
votes
2
answers
12k
views
elasticsearch with kibana displays empty dashboard
I have the following is showing up when I pull up my kibana dashboard:
http://192.168.10.25/#/dashboard
{{dashboard.current.title}}
When I tail my logs I see the following
2014/05/14 13:31:45 [...
- 587
4
votes
1
answer
4k
views
rsyslog - How to flush queue?
I'm using rsyslog to get logs from haproxy and put them into logstash for elasticsearch/kibana.
Everything was working ok, but i've found something strange in rsyslog.
I've found that i've got ...
- 309
8
votes
4
answers
5k
views
How best to monitor logstash?
I've seen this question on the mailing list a few times but haven't had a satisfactory answer.
How best to monitor that the pipeline isn't stuck? Clients -> logstash -> elasticsearch.
Logstash and ...
- 3,002
2
votes
2
answers
2k
views
Can I run elasticsearch on a single server?
I use elasticsearch as part of a Logstash stack, in which all of the components of the stack are installed on the same server.
The purpose of this is to expose application logs to developers for ...
- 3,529
2
votes
1
answer
688
views
Elasticsearch on EC2, webserver on Heroku // EC2 ports only accessible from Heroku
I have an instance of Elasticsearch facing the internet on an EC2.
On the other hand I have my webservice as a Heroku app.
I want to secure the Elasticsearch instance and allow inbound traffic only ...
- 121
4
votes
4
answers
32k
views
Nothing appearing in kibana dashboard
I have installed
Logstash
ElasticSearch
Kibana
on an EC2 instance.
I can visit http://example.com:9200 Here I get
{
"status" : 200,
"name" : "Aleta Ogord",
"version" : {
"number" : "1.1.0"...
- 153
0
votes
1
answer
4k
views
Connection hangs & never connects between ec2 instances over port 9200; fast from outside AWS
I am trying to connect from a web server EC2 instance to an ElasticSearch server ec2 instance. The connection is slow to nonexistent when connecting from the EC2, but very fast when connecting from a ...
- 1,310
6
votes
1
answer
11k
views
ElasticSearch Server Randomly Stops Working
I have 2 ES servers that are being fed by 1 logstash server and viewing the logs in Kibana. This is a POC to work out any issues before going into production. The system has ran for ~1 month and every ...
- 1,393
-2
votes
1
answer
4k
views
Elasticsearch throws Could not reserve enough space for object heap
It seems that Elasticsearch stops working this morning.
When i launch : sudo service elasticsearch start, it says : Starting ElasticSearch Server Done
But when i check the status, it gives me : ...
- 107
0
votes
1
answer
1k
views
Elasticsearch stopped working suddenly
I have deployed Elasticsearch on a development Ubuntu VPS.
We use that VPS only to host API for mobile apps.
Everything worked good till today, elasticsearch stopped working.
I tried to restart it, ...
- 107
0
votes
1
answer
5k
views
ElasticSearch Multiple Indexes and Routing
Currently I have the following setup.
Syslog --> Logstash --> ElasticSearch --> Kibana
Logstash is creating a daily index "/etc/elasticsearch/data/test-elasticsearch/nodes/0/indices/logstash-2014.02....
- 1,393
0
votes
0
answers
2k
views
Output Logstash to Elasticsearch cluster
I have multiple EC2 instances running Logstash and Elasticsearch.
I'd like to output everything into ES cluster.
output {
stdout { debug => true debug_format => "json" }
...
- 1,916
0
votes
1
answer
2k
views
Multiple instances of Logstash + Elasticsearch on AWS
I have setup a Cloudformation template to start an instance where Logstash and Elasticsearch (not embedded) are installed.
I have standard tcp input configured on Logstash and output it to an ...
- 1,916
3
votes
1
answer
5k
views
Logstash Server Log FIle Flooded With "Failed to flush outgoing items" Error Messages
I'm new to Logstash, trying to get it running following the tutorial in "The Logstash Book". On page 44, the guide suggests to tail the Logstash process' main log file called central.log. After ...
- 31