Server Fault is a question and answer site for system and network administrators. It only takes a minute to sign up.
Sign up to join this community
I run an application with fairly chatty logs, which we are forwarding to Splunk. Users are building custom alerts (as well as searches and even dashboards) for themselves.
We are increasingly relying on this forwarder and it is time to begin monitoring the daemon itself.
I'm wondering, if Splunk can generate an alert based on a message matching a certain criteria not arriving for longer than the specified time?
Every forwarder "phones home" at regular intervals. You can search index=_internal for these messages and alert if you don't find them. Look for "phoneHome" (sorry, the exact text escapes me ATM and I'm not at my work computer). Splunk's Monitoring Console will do this for you if you enable the "Missing Forwarders" alert.
