1

I am running a Zabbix server on Ubuntu 16.04 in a location with really bad grid power. Even with a UPS, the Zabbix server frequently goes down. Because the whole building looses power, I have now way of knowing how long the Zabbix server was down.

Management needs to know how long the server was down. How can I read the log files (with a script or single command) to determine the downtime for the Zabbix server?

3 Answers 3

2

If you are talking retrospectively, then assuming reasonable defaults on ubuntu 16.04, you have the following options on (and possibly others... auditd maybe?) to inspect for activity:

  1. systemd journal
  2. log files in /var/log/{auth,kern,syslog}.log etc
  3. the Zabbix server itself

journalctl

I think that persistence of boot logs is disabled on ubuntu, so journalctl -b is probably not that useful. But there is nothing stopping you bucketing the log entries and seeing which periods are "down-time" from all the cron and daemon related logging.

The following will dump out the number of log entries in the systemd journal in hourly chunks going back 1 week. And you can easily see if there was downtime...

START_STRING="1 week ago"
FORMAT="%Y-%m-%d %H:%M:%S"
next_date=$(date +%s -d "$START_STRING")

while [ "$next_date" -le "$(date +%s)" ]; do
  curr_date=$next_date
  next_date=$(date +%s -d "@$((next_date + 3600))")
  curr_data_iso=$(date +"$FORMAT" -d "@$curr_date")
  next_date_iso=$(date +"$FORMAT" -d "@$next_date")
  echo -n "$curr_data_iso  $next_date_iso "
  count=$(journalctl --quiet --no-pager --since "$curr_data_iso" --until "$next_date_iso"  | wc -l )
  echo -n " $count "
        [ $count -gt 100 ] && nummer=100 || nummer=$count
  printf '=%.0s' $(seq 1 $nummer)
        echo ""
done
echo ""

output is something like this

2018-02-05 03:44:26  2018-02-05 04:44:26  15 ===============
2018-02-05 04:44:26  2018-02-05 05:44:26  9 =========
2018-02-05 05:44:26  2018-02-05 06:44:26  18 ==================
2018-02-05 06:44:26  2018-02-05 07:44:26  9 =========
2018-02-05 07:44:26  2018-02-05 08:44:26  9 =========
2018-02-05 08:44:26  2018-02-05 09:44:26  9 =========
2018-02-05 09:44:26  2018-02-05 10:44:26  9 =========
2018-02-05 10:44:26  2018-02-05 11:44:26  121 ====================================================================================================
2018-02-05 11:44:26  2018-02-05 12:44:26  9 =========
2018-02-05 12:44:26  2018-02-05 13:44:26  9 =========
2018-02-05 13:44:26  2018-02-05 14:44:26  9 =========
2018-02-05 14:44:26  2018-02-05 15:44:26  9 =========
2018-02-05 15:44:26  2018-02-05 16:44:26  9 =========
2018-02-05 16:44:26  2018-02-05 17:44:26  9 =========
2018-02-05 17:44:26  2018-02-05 18:44:26  9 =========
2018-02-05 18:44:26  2018-02-05 19:44:26  9 =========
2018-02-05 19:44:26  2018-02-05 20:44:26  12 ============
2018-02-05 20:44:26  2018-02-05 21:44:26  9 =========
2018-02-05 21:44:26  2018-02-05 22:44:26  11 ===========
2018-02-05 22:44:26  2018-02-05 23:44:26  9 =========
2018-02-05 23:44:26  2018-02-06 00:44:26  9 =========
2018-02-06 00:44:26  2018-02-06 01:44:26  9 =========
2018-02-06 01:44:26  2018-02-06 02:44:26  16 ================
2018-02-06 02:44:26  2018-02-06 03:44:26  9 =========
2018-02-06 03:44:26  2018-02-06 04:44:26  9 =========
2018-02-06 04:44:26  2018-02-06 05:44:26  0 =
1
  • Great script! I think I'll doing something different, but along these lines. I think a script to read syslog will give me some accurate downtime totals. Your example has helped me get started. Thanks!
    – zDaniels
    Feb 6, 2018 at 7:15
1

Zabbix monitors itself by default. If it's up, it will have data; if not, it won't. Look for the gap in the data.

2
  • While it is possible to do this, I think I found a faster way of gathering the information instead of manually combing through Zabbix history.
    – zDaniels
    Feb 7, 2018 at 0:58
  • Your solution will give you exact amounts of time. But never underestimate the utility of a graph when presenting something to management. Feb 7, 2018 at 2:50
1

I was able to read the Ubuntu log files using the following command.

grep -E "syslogd:.*(exiting|start)" syslog

This gave me the lines from the log files indicating the date and time that the server shutdown and then booted. For the time being, I'll have to make a spreadsheet with the times to calculate downtime. Maybe I'll write a script at some point to automate this.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .