0

I am looking into the ELK stack, and I mostly got it running. My goal would be to get the Apache2 dashboards running in Kibana, but I am unable to. This is how far I got:

**On the Apache side:*

I am using vhosts_combined log format. I tried to use the apache2 module, but it would not parse the data on the Logstash side.

On the host where Apache2 is running, I installed the filebeat, and I configured it like this: in filebeat.yml I disabled filebeat.inputs, enabled setup.kibana and pointed it to my Kibana host, disabled output.elasticsearch, enabled output.logstash and added, at the end, this:

filebeat.prospectors:
  - paths:
    - /var/log/apache2/access*
    tags: [apache_access]

Then I ran filebeat setup, but I got: Exiting: Template loading requested but the Elasticsearch output is not configured/enabled

So I ran filebeat setup --dashboards and I got confirmation.

On the Logstash side:

On the Logstash host, I put this into the beats.conf file:

input {
  beats {
    port => 5044
  }
}

filter {
  if "apache_access" in [tags] {
    grok {
      match => { "message" => "%{HOSTNAME:vhost}\:%{NUMBER:port} %{COMBINEDAPACHELOG}"}
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

On Kibana:

Now I go to Kibana, and in the Discover tab I see that the data are there and are parsed. However, Apache dashboards are non functional, and I can see:

Could not locate that index-pattern-field (id: apache2.access.remote_ip)

Is there any way I can use those dashboards?

This is the JSON I see in the Discover tab:

{
  "_index": "filebeat-6.6.1-2019.03.07",
  "_type": "doc",
  "_id": "Fbm8V2kBi7AHxLGEc7Po",
  "_version": 1,
  "_score": null,
  "_source": {
    "vhost": "MYSAMPLEHOST.com",
    "agent": "\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0\"",
    "beat": {
      "hostname": "ubuntu",
      "name": "ubuntu",
      "version": "6.6.1"
    },
    "@timestamp": "2019-03-07T10:37:30.160Z",
    "referrer": "\"-\"",
    "clientip": "192.168.56.1",
    "request": "/lib/icons-e6866a5982aa3f74d114acb3a4ae4a72/apple-touch-icon-180x180.png",
    "log": {
      "file": {
        "path": "/var/log/apache2/access.log"
      }
    },
    "bytes": "15302",
    "port": "80",
    "host": {
      "os": {
        "platform": "ubuntu",
        "family": "debian",
        "name": "Ubuntu",
        "codename": "bionic",
        "version": "18.04.2 LTS (Bionic Beaver)"
      },
      "name": "ubuntu",
      "architecture": "x86_64",
      "id": "09be1c222bc742e9bba35b5c29c09036",
      "containerized": false
    },
    "timestamp": "07/Mar/2019:11:37:24 +0100",
    "verb": "GET",
    "httpversion": "1.1",
    "response": "200",
    "source": "/var/log/apache2/access.log",
    "offset": 569899,
    "tags": [
      "apache_access",
      "beats_input_codec_plain_applied"
    ],
    "ident": "-",
    "auth": "-",
    "message": "MYSAMPLEHOST.com:80 192.168.56.1 - - [07/Mar/2019:11:37:24 +0100] \"GET /lib/icons-e6866a5982aa3f74d114acb3a4ae4a72/apple-touch-icon-180x180.png HTTP/1.1\" 200 15302 \"-\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0\"",
    "@version": "1"
  },
  "fields": {
    "@timestamp": [
      "2019-03-07T10:37:30.160Z"
    ]
  },
  "sort": [
    1551955050160
  ]
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.