I am running a Zabbix server on Ubuntu 16.04 in a location with really bad grid power. Even with a UPS, the Zabbix server frequently goes down. Because the whole building looses power, I have now way of knowing how long the Zabbix server was down.

Management needs to know how long the server was down. How can I read the log files (with a script or single command) to determine the downtime for the Zabbix server?

If you are talking retrospectively, then assuming reasonable defaults on ubuntu 16.04, you have the following options on (and possibly others... auditd maybe?) to inspect for activity:

  1. systemd journal
  2. log files in /var/log/{auth,kern,syslog}.log etc
  3. the Zabbix server itself

journalctl

I think that persistence of boot logs is disabled on ubuntu, so journalctl -b is probably not that useful. But there is nothing stopping you bucketing the log entries and seeing which periods are "down-time" from all the cron and daemon related logging.

The following will dump out the number of log entries in the systemd journal in hourly chunks going back 1 week. And you can easily see if there was downtime...

START_STRING="1 week ago"
FORMAT="%Y-%m-%d %H:%M:%S"
next_date=$(date +%s -d "$START_STRING")

while [ "$next_date" -le "$(date +%s)" ]; do
  curr_date=$next_date
  next_date=$(date +%s -d "@$((next_date + 3600))")
  curr_data_iso=$(date +"$FORMAT" -d "@$curr_date")
  next_date_iso=$(date +"$FORMAT" -d "@$next_date")
  echo -n "$curr_data_iso  $next_date_iso "
  count=$(journalctl --quiet --no-pager --since "$curr_data_iso" --until "$next_date_iso"  | wc -l )
  echo -n " $count "
        [ $count -gt 100 ] && nummer=100 || nummer=$count
  printf '=%.0s' $(seq 1 $nummer)
        echo ""
done
echo ""

output is something like this

2018-02-05 03:44:26  2018-02-05 04:44:26  15 ===============
2018-02-05 04:44:26  2018-02-05 05:44:26  9 =========
2018-02-05 05:44:26  2018-02-05 06:44:26  18 ==================
2018-02-05 06:44:26  2018-02-05 07:44:26  9 =========
2018-02-05 07:44:26  2018-02-05 08:44:26  9 =========
2018-02-05 08:44:26  2018-02-05 09:44:26  9 =========
2018-02-05 09:44:26  2018-02-05 10:44:26  9 =========
2018-02-05 10:44:26  2018-02-05 11:44:26  121 ====================================================================================================
2018-02-05 11:44:26  2018-02-05 12:44:26  9 =========
2018-02-05 12:44:26  2018-02-05 13:44:26  9 =========
2018-02-05 13:44:26  2018-02-05 14:44:26  9 =========
2018-02-05 14:44:26  2018-02-05 15:44:26  9 =========
2018-02-05 15:44:26  2018-02-05 16:44:26  9 =========
2018-02-05 16:44:26  2018-02-05 17:44:26  9 =========
2018-02-05 17:44:26  2018-02-05 18:44:26  9 =========
2018-02-05 18:44:26  2018-02-05 19:44:26  9 =========
2018-02-05 19:44:26  2018-02-05 20:44:26  12 ============
2018-02-05 20:44:26  2018-02-05 21:44:26  9 =========
2018-02-05 21:44:26  2018-02-05 22:44:26  11 ===========
2018-02-05 22:44:26  2018-02-05 23:44:26  9 =========
2018-02-05 23:44:26  2018-02-06 00:44:26  9 =========
2018-02-06 00:44:26  2018-02-06 01:44:26  9 =========
2018-02-06 01:44:26  2018-02-06 02:44:26  16 ================
2018-02-06 02:44:26  2018-02-06 03:44:26  9 =========
2018-02-06 03:44:26  2018-02-06 04:44:26  9 =========
2018-02-06 04:44:26  2018-02-06 05:44:26  0 =
  • Great script! I think I'll doing something different, but along these lines. I think a script to read syslog will give me some accurate downtime totals. Your example has helped me get started. Thanks! – zDaniels Feb 6 at 7:15

Zabbix monitors itself by default. If it's up, it will have data; if not, it won't. Look for the gap in the data.

  • While it is possible to do this, I think I found a faster way of gathering the information instead of manually combing through Zabbix history. – zDaniels Feb 7 at 0:58
  • Your solution will give you exact amounts of time. But never underestimate the utility of a graph when presenting something to management. – Michael Hampton Feb 7 at 2:50

I was able to read the Ubuntu log files using the following command.

grep -E "syslogd:.*(exiting|start)" syslog

This gave me the lines from the log files indicating the date and time that the server shutdown and then booted. For the time being, I'll have to make a spreadsheet with the times to calculate downtime. Maybe I'll write a script at some point to automate this.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.