Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

1
vote
0answers
147 views

Filebeat and downstream availability

I read here and there that a broker (like Redis) might not be required in the log pipeline (typically ELK) when Filebeat is used. From Filebeat's official page: [Filebeat] is intelligent enough to ...
0
votes
1answer
2k views

ELK: LogStash to read log files from remote Samba-mapped network drives

I'm new to ELK, and I would like to set up a solution to index Microsoft IIS and applicative .NET logs with ES. I'm aware about different approaches: 1) [app servers: log files ➔ Logstash] ➔ [...
0
votes
1answer
664 views

Elasticsearch Cluster Configuration

Brand new to Elasticsearch. I've been doing tons of reading, but I am hoping that the experts on SO might be able to weigh in on my cluster configuration to see if there is something that I am ...
1
vote
0answers
23 views

elasticsearch creating files but not releasing them until I restart the service

I am running elastcsearch 0.90.5 and recently I have been having issues with the disk running out of space. This is not caused by my indexes increasing in size but by the elasticsearch service ...
0
votes
1answer
3k views

Best practice of adding node(s) to elasticsearch cluster

I'm using Zen Discovery module to connect all nodes together, but when I want to add a new node to the cluster, I have to change every node's discovery.zen.ping.unicast config to add the new node's ip,...
3
votes
1answer
2k views

Logstash integration with AWS Elasticsearch Service

I am using AWS Elasticsearch service to configure Elasticsearch Cluster and there is a separate server where I have installed Logstash 2.1.0 Here is my Logstash sample configuration file :- input { ...
2
votes
1answer
1k views

How to format log data before forwarding them as JSON to elasticsearch?

I log all events on a system to a JSON file via syslog-ng: destination d_json { file("/var/log/all_syslog_in_json.log" perm(0666) template("{\"@timestamp\": \"$ISODATE\", \"facility\": \"$FACILITY\", ...
0
votes
0answers
20 views

Minimum hardware requirements for ElasticSearch in docker [duplicate]

I am developing a search application using elasticsearch and I would like to keep the development cost as low as possible. What are the minimum requirements for elasticsearch to run properly inside a ...
-2
votes
1answer
599 views

Can Logstash, ElasticSearch and Kibana be used to log http requests?

I need something like ODBC logging on Linux. There are some older libraries for Apache and MySQL, but I do not want to use them. I found the ELK stack and installed it. Can Logstash, ElasticSearch ...
1
vote
0answers
83 views

Elasticsearch shuts down with no reason

We're running Elasticsearch 1.7.2 on ubuntu 14.04, the system has enough memory and other services on the server work just fine. But sometimes elastic shutsdown without any thing logged to it's log, ...
1
vote
1answer
419 views

ELK stack on google compute engine

Trying to set up an ELK stack on a google compute engine server I found out that the "click to deploy" solution google is providing is outdated. None the less I tried installing Logstash and Kibana ...
0
votes
1answer
3k views

elasticsearch not creating replica shards

I am banging my head for the past two days on this one. I have setup an elasticsearch cluster of two nodes. The configuration of each node is pretty simple: node.name: "server1" discovery.zen.ping....
3
votes
1answer
448 views

Logstash - Graph IPv4 vs IPv6

I'm looking at starting to graph inbound connections to our various services (web, mail, dns etc) by IP version, ie IPv4 connections versus IPv6 connections. Currently the IPs are being stored in a ...
0
votes
1answer
988 views

Backing Up Old Indices in Elasticsearch

We have a ELK (ElasticSearch-Logstash-Kibana) deployment in which we ship logs via logstash to Elasticsearch Cluster. Indices are created daily. We close indices which are more than 3 days old and ...
0
votes
1answer
141 views

elasticsearch: how to see that warnings are in logs / breakers are being tripped

ES 1.7.x on CentOS Our production ES cluster went down hard. We lost the entire index. Turns out, this had been in the logs for a day or more: New used memory from field ### would be larger than ...
4
votes
1answer
618 views

ElasticSearch snapshot backup sliding expiration - possible?

I'm planning on using the ElasticSearch S3 cloud plugin to create snapshots of our ES cluster. This all looks fairly straight forward but I'm wondering whether its possible integrate it into our ...
0
votes
1answer
2k views

Elasticsearch index created with fieldtype mapping integer, recognized as “string” in Kibana 4

I have created an index in elasticsearch 1.7 like this: curl -XPUT 'http://localhost:9200/test' -d ' { "test" : { "properties" : { "user" : {"type" : "string", "index" : "...
4
votes
3answers
8k views

Foward slash in kibana 3 query

I'm trying to add a query that will match a request that ends with a slash, like this one: n.n.n.n - - [16/Oct/2013:16:40:41 +0100] "GET / HTTP/1.1" 200 25058 "-" "Mozilla/5.0 (iPad; CPU OS 7_0_2 ...
-2
votes
2answers
534 views

CentOs 6.2 server with ElasticSearch crashing multiple times each day

Im running a CentOS 6.2 server 1GB Memory 98 GB Disk Size Elasticsearch version 1.4.2 with around 9000 documents and using JDBC to retrieve the records from a mysql database Over the past few ...
1
vote
0answers
44 views

Experiences with logfile monitoring

In an IT-integrator, I'm currently trying to evangelize the use of log file monitoring of hardware installations. The idea is to use a Elasticsearch/Logstash/Kibana stack (most flexibility, open ...
1
vote
0answers
826 views

How do I update Logstash data to Redshift real-time?

I am looking for a way to build a pipeline for storing the logs data (I use the ELK stack) to AWS Redshift, for analytics. I have gone through the Output plugins for ElasticSearch, and found one for ...
3
votes
0answers
1k views

Curl receives empty repsonse form Elasticsearch server

I have a healthy ES cluster running on ec2. I'm tyring to connect Graylog server to the cluster using unicast, but Graylog receives an empty reponse from the unicast host. Using curl, I can ...
4
votes
4answers
31k views

Nothing appearing in kibana dashboard

I have installed Logstash ElasticSearch Kibana on an EC2 instance. I can visit http://example.com:9200 Here I get { "status" : 200, "name" : "Aleta Ogord", "version" : { "number" : "1.1.0"...
0
votes
0answers
2k views

Output Logstash to Elasticsearch cluster

I have multiple EC2 instances running Logstash and Elasticsearch. I'd like to output everything into ES cluster. output { stdout { debug => true debug_format => "json" } ...
1
vote
1answer
2k views

Elasticsearch Snapshot Restore with Merge

I'm experimenting with Elasticsearch in relation to backups and restoring data. I can back up data into a snapshot using curator no problems. I then physically delete the files related to the index (...
1
vote
1answer
547 views

Why does elasticsearch reports swapping when swapping is disabled?

When trying to disable JVM swapping I disabled paging on a Windows machine and rebooted: When looking at the cluster health via Elastic HQ, I see that the JVM swaps very much anyway: Why is it so? ...
0
votes
1answer
84 views

Opserver Alerting that ElasticSearch Indexes and Shards are in Trouble

We're using Opserver to monitor our ElasticSearch nodes. Recently it began alerting "1 Indexes in trouble" and "3 Shards in trouble". I can't seem to find any other information on these warnings. ...
5
votes
1answer
1k views

Why would a node not have shards?

I am setting up a small elasticsearch cluster and one of the nodes does not get any shards. It works, is part of the cluster - just empty (eu2 in the screenshot of head below): This node is a worker, ...
0
votes
2answers
563 views

Secure elasticsearch but allow remote CRUD

I'm going to run a bit of software on server A to add and remove entries from elasticsearch on server B. How can I allow these remote create and delete operations, but still keep ES secure? Is ...
-1
votes
1answer
261 views

Unable to run logstash config file (permission denied) [closed]

my config file is stored in /etc/logstash/ and I ran the command $ /etc/logstash -f /etc/logstash/logstash.conf as root. However, they told me that permission denied when I tried to do that. ...
2
votes
2answers
3k views

Elasticsearch service often goes down or gets killed

My webserver is running ubuntu 14.10 with elasticsearch 1.5.0 and java 1.7u55 For some reason, the elasticsearch service often goes down, resulting in my website not being available to my users ...
1
vote
0answers
1k views

What is wrong with my logstash configuration?

Background I have two types of log files: output from an ETL process, and then output from a downstream processor. We call these "ETL" and "Processor" logs. The ETL logs are in their own folder ...
4
votes
1answer
2k views

Logstash/elasticsearch stops accepting new data

I've set up a new proof of concept logstash system CentOS 6.6 (on Vmware 5.5) - single CPU VM with 12G RAM allocated Elasticsearch and Logstash installed from RPMs … # rpm -q elasticsearch logstash ...
0
votes
1answer
69 views

Central Logging Options - need flat file as well as elastic search

I would like to have ephemeral ec2 instances push logs to a central flat-file store for archiving and manual perusing, as well has have that data pushed to elastic search. Is there a single agent ...
0
votes
2answers
7k views

ELK stack: Can't load Kibana 4

I have installed Elasticsearch, Logstash, Kibana and Nginx following this guide, exept I used the latest version of everything. https://www.digitalocean.com/community/tutorials/how-to-install-...
3
votes
2answers
1k views

Full data backup to Amazon S3

I have an Ubuntu server hosted on Digital Ocean that has outgrown it's existing backup solution. The relevant parts of the stack I use are Node.js, MongoDB, and Elasticsearch. So far backups have ...
0
votes
1answer
2k views

Logstash is not processing MySQL logs correctly

As you can see, the message is not just the message, but also contains the date and timestamp. . . This is the MySQL Log: /var/log/mysql/error.log 150630 9:01:29 [Warning] Access denied for user '...
1
vote
1answer
1k views

Elasticsearch network.bind_host and network.publish_host

I have a 4gb ram machine that works as a DB server for a ios/webapp. On that machine I only have Elasticsearch and MySql installed. What I want to do now is to only allow connections to ES from two ...
0
votes
0answers
2k views

Unable to fetch mapping. Do you have indices matching the pattern?

I am new to ELK and recently installed ELK on Windows Server 2008 and on Windows 7 machine. But on both the places if I open Kibana I get the message: "Unable to fetch mapping. Do you have indices ...
1
vote
1answer
1k views

Setting memory mapped file limit for Windows?

Hi running Windows Server 2008 R2 Enterprise. My server has 128GB of RAM. I'm using Elasticsearch which is configured with ES_HEAP_SIZE=30GB With Task Manager Private Working Set: 33GB Working Set: ...
1
vote
0answers
734 views

Elasticsearch snapshot / curl error

I'm currently having an issue getting backups for ElasticSearch working correctly via their snapshotting system. Per their instructions, a snapshot repository has to be configured first, using this ...
4
votes
1answer
1k views

Elasticsearch: High traffic on loopback interface

One of our application uses Elasticsearch (1.4.4) as an in-memory cache. The application is a Java webapp deployed on Tomcat 7 with Oracle 1.7. The elasticsearch instance is a one-node setup deployed ...
0
votes
1answer
2k views

Is it possible to access Kibana (Elasticsearch) via ssh tunnel?

I am trying to acces kibana from a remote machine through an SSH tunnel: $ ssh -f user@elastichost -L 5601:elastichost:5601 -N My remote browser just timesout waiting for a response. Is there a ...
3
votes
0answers
2k views

Shutting down elasticsearch completely?

I have found a number of posts about shutting down a node or a cluster of nodes, but what about stopping all elasticsearch processes on a machine? Even after shutting down all nodes (curl -XPOST '...
1
vote
2answers
514 views

Elasticsearch index much larger than the actual size of the logs it indexed?

I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like 2....
3
votes
2answers
6k views

What is the purpose of elasticsearch logs? How to manage them?

As I understand indexes(or data) are being stored in /var/lib/elasticsearch by default, this folder contains nodes with 0 and 1 folders and overall size of these folders is 376M. The logs are ...
1
vote
0answers
588 views

elasticsearch glibc memory corruption

My elastic search clusters keep crashing for no reason. All I can see in the logs is : INFO | jvm 4 | 2015/04/17 19:10:09 | *** glibc detected *** /usr/java/jre1.7.0_04/bin/java: malloc(): ...
1
vote
1answer
2k views

Installing Graylog2 with puppet

I have manged to install Graylog2 from scratch by downloading all the DEB files and making the configurations. Now I try to install Graylog2 with the help of puppet in an environment and had no luck ...
3
votes
0answers
316 views

elasticsearch fails periodically on ubuntu with systemd : how to resolve this?

I am using Elasticsearch on Ubuntu 14.10 and it's periodically going down (entering a failed state). When down, this throws exceptions on my website, users can't search, connect or update database ...
2
votes
0answers
464 views

Benchmarking for DISK I/O Writes Per Second - ElasticSearch and MySQL

During working on our project we needed to decide should we use "ElasticSearch" or "MySQL" to record the audit/trail logs of the system. Searching speed is not an issue here, we needed to see the disk ...