Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [elasticsearch]

Elasticsearch is a flexible and powerful open source, distributed, real-time search and analytics engine.

1
vote
1answer
841 views

Can I use AWS ES be integrated with Graylog2?

I am on a very initial setup and I want to know if I can use ES offered by AWS as a service to the Graylog2? I am quite sure this cannot be accomplished since AWS doesn't offer anything to ...
1
vote
0answers
187 views

Why does syslog create a user.log instead using syslog.log?

I have experienced something a bit weird for me. I have filebeat monitoring my rsyslog (syslog.log) file and sending it to my logstash. I have noticed that after restarting filebeat where syslog is ...
0
votes
1answer
1k views

Nested objects from MySQL to ElasticSearch

I am new to ES and trying to load data from MYSQL to Elasticsearch using logstash jdbc. In my situation I want to use column values as field names, Please see new & hex in output data, I want '...
1
vote
1answer
436 views

rsyslog and elasticsearch: How to configure multiple servers?

We are currently setting some hosts to forward their logs via rsyslog and omelasticsearch to an elasticsearch cluster. The manual for omelasticsearch seems to allow only one server name of the ES ...
4
votes
0answers
2k views

elasticsearch using bulk insert to post JSON file

I have the following JSON file I have used awk to get rid of empty spaces, trailing, next line awk -v ORS= -v OFS= '{$1=$1}1' data.json I have added a create request at the top of my data.json ...
1
vote
1answer
270 views

Field/value extraction with ELK

I have an industrial system producing log files where some of the lines look like this: component1 v1 component2 v2 component3 v3 ... Where vx is a numerical value (eg. 3.14159). I'm running a ...
1
vote
1answer
3k views

Loading field information in Graylog 2.1.x failed with status 500 (Internal Server Error)

I'm working on setting up a Graylog and I'm getting the following error: Loading field information failed with status: cannot GET http://192.168.2.108:12900/system/fields (500) I am on this page: ...
0
votes
1answer
326 views

Logstash only updates S3 access log indices once per hour

I use a combination of Logstash and the AWS Elasticsearch service to index S3 access logs. The logs are collected in an S3 bucket, processed with the Logstash S3 input filter, renamed after they are ...
1
vote
1answer
1k views

elastic's snapshot and restore module repository_exception

I'm using elk-docker and trying to follow Snapshot And Restore | Elasticsearch Reference [2.4] | Elastic and getting following error: # curl --request PUT --data '{ "type": "fs", "settings": {"...
0
votes
1answer
376 views

Encrypting volumes of a mongo/elasticsearch cluster

I have mongodb and elasticsearch cluster. I want to encrypt the volumes associated with all the instances running in the cluster. If I encrypt only one volume among the cluster than the cluster will ...
0
votes
1answer
97 views

Fluentd elasticsearch comprssion

I forward logs from app servers with fluentd (type tail) to elasticsearch server. Do not understand the following: Can I enable for reducing traffic the compression transmitted logs with type tail ...
5
votes
3answers
9k views

Ubuntu 15.04: Elasticsearch service doesn't start

I followed the first part from the tutorial here to install Elasticsearch. I didn't install java because I already have that installed with other java programs running. sudo service elasticsearch ...
0
votes
1answer
292 views

How to troubleshoot LogStash on Windows when there are no reported errors?

I'm setting up LogStash on Windows and made a test launch of logstash.bat from the console to see if it process records from the log4net file. Here is what it reported but I don't see any records in ...
0
votes
1answer
1k views

Elasticsearch stopped working suddenly

I have deployed Elasticsearch on a development Ubuntu VPS. We use that VPS only to host API for mobile apps. Everything worked good till today, elasticsearch stopped working. I tried to restart it, ...
1
vote
1answer
1k views

Why does graylog2 delete old indicies?

According to http://docs.graylog.org/en/1.3/pages/index_model.html, graylog2 deletes old indicies based on the "elasticsearch_max_number_of_indices". Why can't graylog2 allow no. of indicies to grow? ...
1
vote
3answers
502 views

Elasticsearch high RSS memory consumption

I am seeing too high memory consumption in my elasticsearch instance. I set ES_HEAP_SIZE=4g. The starting command for ES starts with: /usr/lib/jvm/java-8-oracle/jre/bin/java -Xms4g -Xmx4g So far so ...
0
votes
1answer
926 views

IIS advanced logging and ELK stack

just built an ELK server on Windows so I'm new to the process. I've read through the docs but am having trouble parsing out my IIS advanced logs, especially x-forwarded-for data as we're behind a load ...
1
vote
0answers
109 views

Delayed synchronization between two AWS ElasticSearch Service instances

I need to synchronize the indexes of two AWS ElasticSearch Service instances once a day. It's really important that the synchronization is NOT real time. The first index (let's call it ES-A) is ...
0
votes
0answers
82 views

ElasticSearch Cluster fails 5 minutes after starting on Azure

I am trying to build an ElasticSearch cluster on Azure. I have done it successfully for testing purposes with 3 VM's under the same virtual network. It functioned very good. Because of my ...
8
votes
4answers
4k views

How best to monitor logstash?

I've seen this question on the mailing list a few times but haven't had a satisfactory answer. How best to monitor that the pipeline isn't stuck? Clients -> logstash -> elasticsearch. Logstash and ...
-1
votes
1answer
742 views

How to use influxdb as a datasource for elasticsearch? [closed]

I have a usecase where I would like to store conversation (chat) history in influxdb but I would like to have search capability of this textual conversation history, for which I would like to use ...
3
votes
1answer
11k views

elk stack error “unable to fetch mapping do you have indices matching the pattern”

i am trying to setup ELK stack with collectd on Ubuntu 16.04 LTS (so pretty much latest version of stack available) kibana is behind nginx proxy (followed this guide https://www.digitalocean.com/...
0
votes
1answer
105 views

ElasticSearch enforce master shard allocation & Improve cross DC write speed

I currently run three Elasticsearch nodes. Each one in a different data center (EU, US, AP) as an AWS EC2 instance. There are 2 replicas per each shard for each index. Writes go only to the EU node. ...
0
votes
1answer
194 views

Unassigned shards in elasticsearch

We have a 0.90 cluster with two elasticsearch nodes. For a while now their status was going yellow often then coming back to green, but now it's red and seems to stay that way. The health is this : { ...
0
votes
1answer
311 views

Logstash Grok Filter

I'm sure there is a simple solution to this but I'm new to working Logstash. The filter I'm trying to apply is for some firewall logs and I have a field that will either have a single or two values ...
0
votes
1answer
258 views

Can I setup logstash with windows generated certs instead of openssl?

Going through this tutorial. They use openssl to generate certs to use with logstash ie this command: sudo openssl req -subj '/CN=ELK_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -...
0
votes
0answers
223 views

Best practice/way to manage Elasticsearch cluster nodes in EC2 cloud with Ansible

I have to deploy cluster for production environment in EC2 cloud and want to be able to: 1. Remove/Add nodes from/to cluster 2. Assign roles to nodes (dedicated master node, data node, and client ...
1
vote
2answers
654 views

How to balance elasticsearch (java) on a very powerful machine?

On a 16 core XEON class, 128GB RAM RHEL server I want to deploy elasticsearch. What is preferable performance-wise? Have a huge elasticsearch process to use all resources on the native host? Break ...
12
votes
4answers
54k views

Ubuntu uninstall elasticsearch

I installed elasticsearch.90.7 with a deb file in ubuntu. I tried to uninstall elasticsearch.90.7 with this command: sudo apt-get --purge autoremove elasticsearch And then I downloaded elasticsearch-...
2
votes
1answer
86 views

Log file in Topbeat

I am new to Elasticsearch and Topbeat tools. When I start the topbeat service and look through my browser by typing localhost:9200/topbeat-*/_search?pretty, I get data about my system. But I want ...
0
votes
2answers
2k views

Collectd --> Elasticsearch if remote-host can't connect to central elasticsearch

Goal Central storage and way to analyze performance numbers: cpu load ram usage ... Current strategy I would like to implement a setup like this: collectd logstash elasticsearch kibana Like ...
0
votes
1answer
171 views

Field type not being honor in Kibana/Elasticsearch

I'm using a logstash plug-in for a Java app using logback to forward logs on to my logstash server. I've setup a filter definition as follows: input { tcp { port => 2856 codec => ...
1
vote
0answers
645 views

object mapping for [doc.awsRegion] tried to parse field [awsRegion] as object, but found a concrete value

I am using Logstash Dynamodb PLugin to index data to be able to query it. Everything was running pretty smoothly, until one of my teammates started messing with the schema, and broke it. We can still ...
1
vote
1answer
650 views

Sending docker logs to logstash

I have a number of CoreOS servers on Amazon AWS and would like to collect events or logs from them and forward them onto my ELK stack provider logz.io. Being a little new to the ELK stack, I'm a ...
1
vote
0answers
180 views

Redis creates many events in logstash

I'm doing some testing for a POC with logstash. When using redis to buffer messages it appears they stay in redis and continually get added to elasticsearch as new events. For example, if I restart ...
0
votes
1answer
271 views

Elasticsearch ec2 discovery getting wrong IP?

I am running elasticsearch 1.4.3 as a worker-node/client (master=false, data=false) on AWS and I'm trying to get it to join a cluster. My elasticsearch.yml looks like this: cluster-name: ...
3
votes
3answers
1k views

Must all nodes of an Elasticsearch cluster be identical?

In an Elasticsearch cluster, must all nodes be identical i.e. having the same: Operating System Amount of RAM CPU speed and number of cores etc. or is it allowed for the nodes to be different? (...
0
votes
2answers
63 views

ElasticSearch cluster move to new node type

I have a cluster of 8 nodes in ec2. 4 per AZ with cluster.routing.allocation.awareness.attributes: aws_availability_zone I want to migrate all the shards to a different node type. So I set up to new ...
0
votes
0answers
2k views

F5 High Speed Logging and Logstash GROK

Im trying to format logs straight from our F5 using HSL and Logstash. I've copied the example configuration exactly from the site as it contains the majority of the information I wanted logstash ...
0
votes
1answer
865 views

How to add Elasticsearch as a service in Ambari

I was wondering to know how can I add Elasticsearch as a service to Ambari portal and monitor it through that portal? ... I know it's doable via Ambari Stacks, but I don't know how doing it. I found ...
0
votes
0answers
1k views

Sending PostGreSQL log data to Logstash using Packetbeat

I am trying to configure sending pgsql log data to elasticsearch. The architecture is like this: packetbeat -> logstash -> elasticsearch. After running packetbeat, logstash log says: :message "Beats ...
0
votes
1answer
89 views

Cassandra JMX doc pointers

Any good starting URL/pointers for reading up on what could be monitored through JMX on cassandra 2.0? Think off pull data into an existing ELK stack from JMX on new Cassandra clusters, but never ...
1
vote
0answers
213 views

Visualize multiline ruby exceptions in kibana 4

I have setup the latest version of Kibana4 ElasticSearch stack. The logs are being pooled from remote app sources which are running on Ruby. I want to search for Multi Line exceptions created by ...
2
votes
1answer
817 views

ElasticSearch container on CoreOS with persistent storage

I'm trying to run an ElasticSearch cluster on Docker and CoreOS [stable] on DigitalOcean. To get persistent storage working, I tried pretty much every option and running out of ideas. Pretty much ...
1
vote
2answers
1k views

Nginx reverse proxy for elasticsearch - inserting filters

I've got an elasticsearch instance, with an Nginx reverse proxy sat in front of it, implementing a URL level access control mechanism - approximately as described in this article This is giving me ...
0
votes
2answers
3k views

Kibana: No Living connections

I have ES 1.5.2 cluster with Kibana 4.0.1. This consists of 4 data/master nodes along with 4 client nodes. When I'm trying to load kibana sometimes it takes longer time to load or sometimes I need to ...
0
votes
2answers
875 views

Logstash filtering syslog by host group?

I've got an Elasticsearch/Logstash/Kibana instance running, which I'm merrily stuffing with syslogs from a variety of hosts. Having built it to scale - with multiple logstash syslogd listeners, and ...
0
votes
1answer
808 views

ElasticSearch Delayed Indexing

I currently have the following setup: syslog-ng servers --> Logstash --> ElasticSearch The syslog-ng servers are load balanced and write to a SAN location where Logstash just tails the files and ...
1
vote
0answers
147 views

Filebeat and downstream availability

I read here and there that a broker (like Redis) might not be required in the log pipeline (typically ELK) when Filebeat is used. From Filebeat's official page: [Filebeat] is intelligent enough to ...
0
votes
1answer
2k views

ELK: LogStash to read log files from remote Samba-mapped network drives

I'm new to ELK, and I would like to set up a solution to index Microsoft IIS and applicative .NET logs with ES. I'm aware about different approaches: 1) [app servers: log files ➔ Logstash] ➔ [...