Questions tagged [elasticsearch]
Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.
379
questions
0
votes
1
answer
2k
views
Logstash is not processing MySQL logs correctly
As you can see, the message is not just the message, but also contains the date and timestamp.
.
.
This is the MySQL Log: /var/log/mysql/error.log
150630 9:01:29 [Warning] Access denied for user '...
- 9,590
1
vote
1
answer
2k
views
Elasticsearch network.bind_host and network.publish_host
I have a 4gb ram machine that works as a DB server for a ios/webapp. On that machine I only have Elasticsearch and MySql installed.
What I want to do now is to only allow connections to ES from two ...
- 2,456
0
votes
0
answers
2k
views
Unable to fetch mapping. Do you have indices matching the pattern?
I am new to ELK and recently installed ELK on Windows Server 2008 and on Windows 7 machine.
But on both the places if I open Kibana I get the message: "Unable to fetch mapping. Do you have indices ...
- 8,936
1
vote
1
answer
3k
views
Setting memory mapped file limit for Windows?
Hi running Windows Server 2008 R2 Enterprise. My server has 128GB of RAM.
I'm using Elasticsearch which is configured with ES_HEAP_SIZE=30GB
With Task Manager
Private Working Set: 33GB
Working Set: ...
- 3,506
1
vote
0
answers
970
views
Elasticsearch snapshot / curl error
I'm currently having an issue getting backups for ElasticSearch working correctly via their snapshotting system. Per their instructions, a snapshot repository has to be configured first, using this ...
4
votes
1
answer
2k
views
Elasticsearch: High traffic on loopback interface
One of our application uses Elasticsearch (1.4.4) as an in-memory cache. The application is a Java webapp deployed on Tomcat 7 with Oracle 1.7. The elasticsearch instance is a one-node setup deployed ...
- 4,109
3
votes
0
answers
2k
views
Shutting down elasticsearch completely?
I have found a number of posts about shutting down a node or a cluster of nodes, but what about stopping all elasticsearch processes on a machine? Even after shutting down all nodes (curl -XPOST '...
- 181
1
vote
2
answers
985
views
Elasticsearch index much larger than the actual size of the logs it indexed?
I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like 2....
- 4,109
5
votes
2
answers
7k
views
What is the purpose of elasticsearch logs? How to manage them?
As I understand indexes(or data) are being stored in
/var/lib/elasticsearch
by default, this folder contains nodes with 0 and 1 folders and overall size of these folders is 376M.
The logs are ...
- 225
1
vote
0
answers
650
views
elasticsearch glibc memory corruption
My elastic search clusters keep crashing for no reason.
All I can see in the logs is :
INFO | jvm 4 | 2015/04/17 19:10:09 | *** glibc detected *** /usr/java/jre1.7.0_04/bin/java: malloc(): ...
- 814
1
vote
1
answer
2k
views
Installing Graylog2 with puppet
I have manged to install Graylog2 from scratch by downloading all the DEB files and making the configurations. Now I try to install Graylog2 with the help of puppet in an environment and had no luck ...
CommunityBot
- 1
3
votes
0
answers
332
views
elasticsearch fails periodically on ubuntu with systemd : how to resolve this?
I am using Elasticsearch on Ubuntu 14.10 and it's periodically going down (entering a failed state).
When down, this throws exceptions on my website, users can't search, connect or update database ...
CommunityBot
- 1
2
votes
0
answers
590
views
Benchmarking for DISK I/O Writes Per Second - ElasticSearch and MySQL
During working on our project we needed to decide should we use "ElasticSearch" or "MySQL" to record the audit/trail logs of the system. Searching speed is not an issue here, we needed to see the disk ...
- 121
2
votes
0
answers
1k
views
Using database for storing text files and images [closed]
I need to build a image/static file cache which would host maybe few hundred up to thousand static files from 1kb to ~200kb. I'd need super fast reads (I'm using redis, but it's one threaded and I'm ...
- 4,970
0
votes
2
answers
368
views
elasticsearch-river-jdbc inserting duplicate record in mysql db
Sorry I am newbie to Elasticsearch and I am using elasticsearch-river-jdbc to connect to myql database,everything is working fine except for the fact that everytime it run as per schedule its ...
CommunityBot
- 1
0
votes
1
answer
628
views
Is it generally a bad idea to run MongoDB and ElasticSerach on the same nodes
Simple question: is it stupid to run mongodb and elasticsearch on the same nodes within a cluster of three if you take care of mongodb's memory hunger using cgroups and ignore the aspect of concurring ...
- 32.8k
1
vote
1
answer
3k
views
How can I add basic auth to an apache proxy while using CORS headers?
Background
I use grafana for displaying graphs of our server-metrics. Grafana is a JS-app which in our case is getting the data from graphite and storing search-queries in elasticsearch. All three ...
- 182
6
votes
2
answers
1k
views
Sending Subversion logs to Logstash
My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own ...
- 713
2
votes
0
answers
819
views
ElasticSearch major slowdown upon big aggregations
My goal is to use ElasticSearch v1.3.2 for analyzing product cross-sales, so I need to filter for the receipts of interest (having an alcoholic product for example) and finding top selling products of ...
- 54k
1
vote
2
answers
3k
views
Elasticsearch as memcache replacement
I am looking into replacing memcached with elasticsearch. Basically, I need to reduce my software stack and since I will be using elastic for searching, I was thinking of creating an index that will ...
CommunityBot
- 1
1
vote
0
answers
348
views
How parse a multiple objects serialized in log with logstash and grok
I'm parsing some log files with grok, but I have the following trace:
[es.org.some.Object[
attr1=53212
searchTags=[love, rock]
]]
[es.org.some.Object[
attr1=54612
searchTags=[love, peace, rock]
...
CommunityBot
- 1
6
votes
0
answers
409
views
Is there disk based file system with dedicated memory caching for Linux? [closed]
We have two (or more sometimes) applications battling for the OS page cache. In particular ElasticSearch is very dependent on the OS page cache. To the point that cache misses can make or break ...
- 228
4
votes
4
answers
546
views
Securing elasticsearch
There appears to be a serious lack of information on this despite the fact that as installed elasticsearch is extremely exploitable.
My main fear in using it is that as a non-expert I have no idea ...
- 3,002
7
votes
3
answers
7k
views
Secure logstash and elasticsearch
I'm considering running logstash on my prod server (simple install. http://logstash.net/docs/1.1.13/tutorials/getting-started-simple) and set kibana to access logs.
My concern is: how to secure my ...
- 3,002
3
votes
1
answer
3k
views
How to rename and join an elasticsearch server preserving data?
I tested elasticsearch and decided to make a cluster. To this I renamed the node and defined a cluster
cluster.name = mycluster
node.name = firstnode
When restarting the changes are taken into ...
- 3,657
2
votes
2
answers
12k
views
elasticsearch with kibana displays empty dashboard
I have the following is showing up when I pull up my kibana dashboard:
http://192.168.10.25/#/dashboard
{{dashboard.current.title}}
When I tail my logs I see the following
2014/05/14 13:31:45 [...
- 146
0
votes
1
answer
163
views
Access the JVM section of elasticsearch through the monitoring API
How I can access the jvm section of the node stats API ?
I've read about it in the docs - monitoring individual nodes
I'm doing a call to
/_nodes/stats?pretty=true
but don't have the jvm section ...
- 76
0
votes
1
answer
3k
views
Is it possible to do autoscaling for elasticsearch in AWS at peak load?
I'm new to ElasticSearch but I'm already using ElasticSearch on my production site.
I installed ElasticSearch in one EC2 instance. After using ElasticSearch for several month, now currently the CPU ...
CommunityBot
- 1
0
votes
1
answer
244
views
Statistical information regarding scaling ElasticSearch [duplicate]
I have a set-up with several Logstash nodes sending input to ElasticSearch, and have a kibana server which allows me to visualize this.
The current infrastructure is pretty simplistic and is on ...
- 84.4k
1
vote
0
answers
2k
views
logstash / kibana can't connect to instance
I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries.
However.. today when I went to my logstash page, I see ...
- 307
0
votes
1
answer
971
views
Elasticsearch River JDBC - Avoiding 'SELECT *'
I'd like to syncronize some MySQL tables to an ElasticSearch index, you know, for search.
But almost every examples I can find show a big bad fat "SELECT * FROM", it's OK for an example, but I'm in ...
- 251
0
votes
1
answer
273
views
Monitoring solution that doesn't duplicate source data?
I need to analyze data stored in multiple databases. Each of them contains a few TB of logs and sensor data.
If I use Splunk or ElasticSearch/Kibana, I see 2 solutions:
Batch import everything
Write ...
0
votes
1
answer
12k
views
Configure Nginx, Kibana, Elasticsearch
My setup
Nginx + Kibana - same box different domain, sub-domain (nginx - example.com, kibana - kibana.example.com)
Elasticsearch - 192.168.100.31
I have used the config defined below. Replaced 127....
- 133
13
votes
1
answer
37k
views
Elasticsearch is using way too much disk space
I have a CentOS 6.5 server on which I installed Elasticsearch 1.3.2.
My elasticsearch.yml configuration file is a minimal modification of the one shipping with elasticsearch as a default. Once ...
CommunityBot
- 1
-1
votes
1
answer
795
views
How do I know how the hacker gained access to my machine? [duplicate]
I own a dedicated machine and today I got an email from my host saying that my server will be blocked if I don't stop abusing the network. I am not a spammer or hacker. Someone was using my machine to ...
CommunityBot
- 1
2
votes
1
answer
1k
views
ES fills up heap slowly over time and hangs at 14GB, while biggest index is 164MB?
I have an issue with Elasticsearch, at some times, it tries to run GC continously since this one is unable to free because heap size set to 14GB (min and max) is said to be completely allocated:
(...)...
- 858
2
votes
2
answers
2k
views
Can I run elasticsearch on a single server?
I use elasticsearch as part of a Logstash stack, in which all of the components of the stack are installed on the same server.
The purpose of this is to expose application logs to developers for ...
- 3,519
1
vote
1
answer
3k
views
Elasticsearch Cluster State immediately goes Yellow after first index created
I run a number of standalone Logstash servers to allow review of log files from web application servers.
One of these recently reported a Yellow cluster state due to unassigned shards. This is a ...
- 3,519
1
vote
0
answers
3k
views
JVM process resident set size "equals" max heap size, not current heap size
After a few reading about jvm memory (here, here, here, others I forgot...), I am expecting the resident set size of my java process to be roughly equal to the current heap space capacity. That's not ...
CommunityBot
- 1
1
vote
1
answer
5k
views
Graylog2 server cannot connect to Elasticsearch
I try to follow Digital Ocean manual except using last versions of components.
My elasticsearch.yml:
cluster.name: graylog2
network.bind_host: localhost
network.publish_host: localhost
script....
- 26
2
votes
3
answers
3k
views
sudden peak in cpu usage
We are running a 4 node/machine elastic search cluster on 12 core, 96gb RAM, 4 spinning disk machines. under normal operation most cpu usage is user and around 5-10%. Every few days, one of the ...
- 5,941
1
vote
1
answer
5k
views
How do I make a graceful shutdown (with zero replicas)?
I am running a three node (elasticsearch) cluster currently with index.number_of_replicas: 0. I'd like to bring one of the node down for maintenance. Do I have any options to do this without ...
CommunityBot
- 1
0
votes
1
answer
4k
views
Numerical range in Kibana 3 not working
My tomcat access logs have last field as "%D - Time taken to process the request, in millis".
I am trying to filter the logs which took more than some number of milliseconds, but it is showing all the ...
- 691
2
votes
1
answer
688
views
Elasticsearch on EC2, webserver on Heroku // EC2 ports only accessible from Heroku
I have an instance of Elasticsearch facing the internet on an EC2.
On the other hand I have my webservice as a Heroku app.
I want to secure the Elasticsearch instance and allow inbound traffic only ...
CommunityBot
- 1
5
votes
2
answers
2k
views
How to secure an Internet-facing Elastic Search implementation in a shared hosting environment?
(Originally asked on StackOverflow, and recommended that I move it here)
I've been going over the documentation for Elastic Search and I'm a big fan and I'd like to use it to handle the search for my ...
CommunityBot
- 1
3
votes
1
answer
3k
views
Using Logstash as shipper?
We are shipping logs from servers and using THE logstash on each server for shipping.
So we read logs from the glob "/root/Desktop/Logstash-Input/**/*_log".
input {
file{
...
- 149
0
votes
0
answers
356
views
indexing a ~3gb database into elasticsearch: how?
I have a new database of 2 million records (around 3gb when dumped json) produced every several days. I want to quickly put it into ElasticSearch.
That's what I do right now:
Create a new index and ...
- 203
1
vote
1
answer
2k
views
Iptables config breaks Java + Elastic Search communication
I am trying to set up a firewall for a server hosting a java app and ES. Both are on the same server and communicate to each other. The problem I am having is that my firewall configuration prevents ...
- 133
1
vote
1
answer
5k
views
Unable to configure Nginx as a reverse proxy for elasticsearch on another server
I'm trying to use Kibana on a server A to access elasticsearch on a server B.
elasticsearch is protected with a basic auth authentication on server B.
Kibana is also protected with basic ...
- 1,393
2
votes
2
answers
6k
views
PHP - does curl run asynchroniously on my system? [closed]
I'm building a web app. I have a database of books indexed in ElasticSearch and REST API written in PHP.
In the app, there's a search box, where I type a name of the book and JS script calls the ...
- 5,173