0

I am building log analyzer for production environment. My scenario is, i need to setupelasticsearch,Logstash and kibana on one centos7 server which is going to use ELK server another one is apache server(remote server).

I have configured following like

(i).ELK server - elasticsearch,kibana ,logstash with nginx proxy

(ii).Application server(apache server)- installed beats on apache server

(i).ELK server configuration

Elasticsearch Configuration

vi /etc/elasticsearch/elasticsearch.yml

network.host: localhost

http.port: 9200

Kibana Dashboard Configuration

vi /etc/kibana/kibana.yml

server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]

Nginx Configuration:

server {
listen 80;
server_name 172.xx.xx.xx;
location / {
    proxy_pass http://localhost:5601;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
  }
}

Logstash Configuration

cat /etc/logstash/conf.d/02-beats-input.conf

input {
  beats {
  port => 5044
 }
}

I have configured filter section too..

cat /etc/logstash/conf.d/30-elasticsearch-output.conf

output {
 elasticsearch {
  hosts => ["localhost:9200"]
  sniffing => true
  manage_template => false
  index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
 }
}

ii) Application server Beats configuration

I have installed beats in application server and ships logs with logstash.

Filebeat configuration:

cat /etc/filebeat/filebeat.yml

output.logstash:
   #the logstash hosts
hosts: ["172.xx.xx.xx:5044"]

here, i am not sure how to load filebeat index template, index and Dashboard in kibana. When i try to run "filebeat setup" command from apache server, i got following error.

[root@webserver ~]# filebeat setup
Exiting: Index management requested but the Elasticsearch output is not configured/enabled.

I know if i configure filebeat with elasticsearch output, i can load indexes directly to kibana. But i don't want apache server directly contact with elasticsearch.

Can someone suggest how can i load "index, index template and dashboards" directly from apache server to kibana while configured beats output to logstash.

I have searched lot of sites, but haven't get any answers. Is there any other alternative method(like import method) to load apache server filebeat indexe in kibana and elk server.

I am eager to wait someone give solution for this.

Thanks

Kumar

2 Answers 2

0

First, take a look at how events get from the source server into ElasticSearch.

[App-Server --> Log-file --> Beats] --> [Logstash --> ElasticSearch].

The Apache server, the app server, has no way to talk to ElasticSearch in this configuration. Therefore, the beats commands to set up the index, template, and dashboards won't work from there. However, your architecture is not set up to assume Beats! It assumes Logstash is the one sending events into ElasticSearch. Kibana can handle that just fine.

If you go into Kibana's management and then index patterns, it will bring up a list of your current indices. With your beats to logstash configuration, you should have some. Set up a wildcard for the indices you want to search.

2
  • Thanks for your reply..I have done configurations as per your guide and It's working fine.But i have faced another issues. when i configure filebeat and metricbeat in apache server, it bring up new indices on ELK server. In same case, auditbeat doesn't bring any indices to ELK server. Is there any other method to load auditbeat indices from client machine with logstash configuration.
    – Kumar
    Jun 3, 2020 at 14:59
  • Also I would like to setup custom index pattern on kibana. Like i have done filebeat configuration through logstash in Apacheserver(Development),Apacheserver(QA testing) and Apacheserver(Production server). But those server logs are getting same filebeat index pattern. Any way to separate both logs from three servers ?
    – Kumar
    Jun 3, 2020 at 15:06
0

Try running: filebeat -e instead of: filebeat setup

1
  • This is an old question that already has an accepted answer. Has something changed in the system that your answer may be more valid today than the answer that was already accepted? If so, please give some more information to show how and why. Dec 22, 2023 at 0:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .