Dose anyone know if it is possible to collect ESX Performance Stats in LogStash/ELK stack? looking to see if I can collect ESXTOP data for use in the ELK stack.
1 Answer
Here's an example input, you would just need rsyslog or a logging tool to send logs to the input port (1514 in this example) and then filter the data:
Input
input {
tcp {
type => "VMware"
port => "1514"
}
}
Filter
filter {
if "VMware" in [tags] {
multiline {
pattern => "-->"
what => "previous"
}
grok {
break_on_match => true
match => [
"message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOGPROG:syslog_program}: (?<message-body>(?<message_system_info>(?:\[%{DATA:message_thread_id} %{DATA:syslog_level} \'%{DATA:message_service}\'\ ?%{DATA:message_opID}])) \[%{DATA:message_service_info}]\ (?<syslog_message>(%{GREEDYDATA})))",
"message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOGPROG:syslog_program}: (?<message-body>(?<message_system_info>(?:\[%{DATA:message_thread_id} %{DATA:syslog_level} \'%{DATA:message_service}\'\ ?%{DATA:message_opID}])) (?<syslog_message>(%{GREEDYDATA})))",
"message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOGPROG:syslog_program}: %{GREEDYDATA:syslog_message}"
]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "YYYY-MM-ddHH:mm:ss,SSS" ]
timezone => "UTC"
}
mutate {
replace => [ "@source_host", "%{syslog_hostname}" ]
}
mutate {
replace => [ "@message", "%{syslog_message}" ]
}
if "Device naa" in [message] {
grok {
break_on_match => false
match => [
"message", "Device naa.%{WORD:device_naa} performance has %{WORD:device_status}%{GREEDYDATA} of %{INT:datastore_latency_from}%{GREEDYDATA} to %{INT:datastore_latency_to}",
"message", "Device naa.%{WORD:device_naa} performance has %{WORD:device_status}%{GREEDYDATA} from %{INT:datastore_latency_from}%{GREEDYDATA} to %{INT:datastore_latency_to}"
]
}
}
if "connectivity issues" in [message] {
grok {
match => [
"message", "Hostd: %{GREEDYDATA} : %{DATA:device_access} to volume %{DATA:device_id} %{DATA:datastore} (following|due to)"
]
}
}
if "WARNING" in [message] {
grok {
match => [
"message", "WARNING: %{GREEDYDATA:vmware_warning_msg}"
]
}
}
}
}
-
if the ESX host is already syslogging to a syslog server and from there Logstash is picking it up, will this work in that context? Mar 5, 2015 at 20:54
-
Yes, you can use any type of logger just as long as you set the port that logstash will expect the data on and set udp/tcp if it's an option. Try this to verify log data on your receiving machine:
telnet <hostname> <port>
Mar 5, 2015 at 21:04 -
perfect, I will try this out and mark answered once I get it working. Mar 5, 2015 at 21:10
-
Hope it works, you may have to play with the filter if there are parts that aren't on the mark. grokparsefailure will be something you see if the filter failed Mar 5, 2015 at 21:11
-