1

We are running FreeNAS (which is built on FreeBSD) to run our data storage systems. It is running on an APC Smart-UPS 750VA X.

On a couple of occasions, I have been alerted to the fact that the server is down by our monitoring systems. After a few minutes the server is back up and running no problems.

When I run last I can see that the server has just booted and checking /var/log/messages I can see it has run through all of the boot process however I can't see any panics or any reason for it shutting down. It literally goes from being fine to outputting boot information.

So this has led me to wonder if its a power outage that is causing this to happen but how can I determine for sure this is the case? I guess getting an interactive card for the APC UPS and hooking it up to the network would be one way... Any other way of me finding out right now why this has happened?

  • No detail on the hardware, whether you have dual power supplies, whether a UPS signal cable is connected, other items in the environment... You shouldn't be asking the internet about your local site's power or hardware issues. – ewwhite Nov 16 '14 at 13:50
  • @ewwhite I'm not asking about my local site's power or hardware issues - I'm asking how I can determine if there are local site power or hardware issues. – Chris Nov 17 '14 at 12:47
  • Which only YOU can tell us! We can't tell you if you're having power issues. – ewwhite Nov 17 '14 at 13:35
7

I think there are a few obvious solutions to finding out more:

  • Hook up an additional device to the same UPS output. If both devices reboot simultaneously, that would be evidence of a power issue.
  • Hook up some equipment that can monitor the actual voltage output by the UPS output and record changes over time.
  • Seems like this could be the answer and I need to hook up some more equipment to monitor the UPS - here goes spending more money :P – Chris Nov 17 '14 at 11:43
  • @Chris The first of my two suggestions doesn't require specialized equipment, so there might be a chance you have something around, which would fulfill that role. All it requires is a device that logs the time of each boot. The second of my suggestions require specialized equipment, but it would also provide more accurate answers. – kasperd Nov 17 '14 at 13:20
2

Your machine can't really tell what happened in a power outage: those electrons just stop showing up. The UPS might know (if you're losing power, as opposed to a flaky power supply or something) but I don't think you have much hope of the server being able to tell you.

  • Thanks for your message. As it is a smart UPS, it is connected to the server and thus we do get notifications when the system switches to battery. So I know the UPS battery wasn't low - I'm just worried the UPS isn't working quite right when there's just a small flicker in power. – Chris Nov 17 '14 at 11:41
2

FreeBSD has a great port named sysutils/apcupsd intended to interact with APC smart-UPSes.

Link your UPS with usb-cable to the host. Edit /usr/local/etc/apcupsd/apcupsd.conf:

###
UPSCABLE usb
UPSTYPE usb
DEVICE 
POLLTIME 60
LOCKFILE /var/spool/lock
BATTERYLEVEL 15
MINUTES 10
TIMEOUT 0
ANNOY 300
ANNOYDELAY 60
KILLDELAY 120
UPSCLASS standalone
STATTIME 60
STATFILE /var/log/apcupsd.status
###

That config means the next behaviour:

  • when power is lost for less than ANNOYDELAY seconds, UPS just goes on battery with no signal

  • after ANNOYDELAY seconds UPS begin to beep.

  • when (accu level becomes lower than BATTERYLEVEL percent) OR (estimated time on battery is less than MINUTES), apcupsd will wait for KILLDELAY seconds and begin shutdown -h now process.

  • after that UPS will power off the load and goes into hybernation.

  • when power is back, UPS power on the load and, if it is configured to boot after power loss, it will be launched normally and cycle is complete.

  • Interestingly we already have this running and we also have email notifications when the battery gets low/ goes onto battery/ comes back online and none of this happened! Appreciate your suggestion though. – Chris Nov 17 '14 at 11:42
  • Smart-UPS have his own EEPROM and can be configured in some way. Sometimes UPS and apcupsd settings conflicting with unpredictable consequencies. – Kondybas Nov 17 '14 at 12:28
  • @Chris There's also a log of UPS events... – ewwhite Nov 17 '14 at 13:21
2

I don't feel like you've done the bare minimum of troubleshooting here. This has become a bad question because of the scare details presented.

  • Maybe your server is crashing.
  • What type of server hardware are you using?
  • Do you have out-of-band management on this server? (ILO, IPMI, DRAC) If so, what does it say?
  • How frequently is this happening?
  • Do any of your other devices in the same room reset? Switches? Other servers? What else is impacted?
  • Where is your monitoring system located in relation to the server, since it's clearly not having problems?

Obvious thing to do...

  • Connect the APC black serial cable to your UPS and your FreeNAS server.
  • Look at the logs of your apcupsd service. (/var/log/serviceapcupsd.events in Linux)

(easy)

2014-10-27 05:18:35 -0400  Power failure.
2014-10-27 05:18:38 -0400  Power is back. UPS running on mains.
2014-10-27 13:20:22 -0400  Power failure.
2014-10-27 13:20:25 -0400  Power is back. UPS running on mains.
2014-10-29 08:00:51 -0400  Power failure.
2014-10-29 08:00:54 -0400  Power is back. UPS running on mains.
2014-10-29 08:02:13 -0400  Power failure.
2014-10-29 08:02:16 -0400  Power is back. UPS running on mains.
2014-11-01 10:05:41 -0400  Power failure.
2014-11-01 10:05:44 -0400  Power is back. UPS running on mains.
  • Check your battery status. They last 3-5 years. If older, it's not holding your power load under transfer to battery power.
  • Check your UPS sensitivity to power fluctuations. They default to HIGH. I usually change this to LOW in crappy environments.
  • Does your FreeNAS server have dual power supplies? (if not, it should) Plug one into the wall to bypass the UPS and eliminate the UPS as a cause. This can also help logging if the server is rebooting.
  • Upgrade your UPS firmware.
  • Upgrade your server's firmware.
  • Call APC support.

(also easy)

[root@General /var/log]# apcaccess status localhost:3551
APC      : 001,052,1316
DATE     : 2014-11-17 08:24:55 -0500
HOSTNAME : General
VERSION  : 3.14.10 (13 September 2011) redhat
UPSNAME  : UPS_IDEN
CABLE    : Custom Cable Smart
DRIVER   : APC Smart UPS (any)
UPSMODE  : Stand Alone
STARTTIME: 2014-09-24 14:14:05 -0400
MODEL    : SMART-UPS 2200 RM XL
STATUS   : ONLINE LOWBATT
LINEV    : 119.6 Volts
LOADPCT  :  60.8 Percent Load Capacity
BCHARGE  : 100.0 Percent
TIMELEFT :   3.0 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
MAXLINEV : 120.9 Volts
MINLINEV : 118.9 Volts
OUTPUTV  : 119.6 Volts
SENSE    : High
DWAKE    : 000 Seconds
DSHUTD   : 180 Seconds
DLOWBATT : 02 Minutes
LOTRANS  : 103.0 Volts
HITRANS  : 132.0 Volts
RETPCT   : 000.0 Percent
ITEMP    : 44.5 C Internal
ALARMDEL : 5 seconds
BATTV    : 56.1 Volts
LINEFREQ : 60.0 Hz
LASTXFER : No transfers since turnon
NUMXFERS : 11
XONBATT  : 2014-11-15 09:05:48 -0500
TONBATT  : 0 seconds
CUMONBATT: 44 seconds
XOFFBATT : 2014-11-15 09:05:52 -0500
SELFTEST : NO
STESTI   : 336
STATFLAG : 0x07000048 Status Flag
DIPSW    : 0x00 Dip Switch
REG1     : 0x00 Register 1
REG2     : 0x00 Register 2
REG3     : 0x00 Register 3
MANDATE  : 11/05/05
SERIALNO : QS0545111716
BATTDATE : 11/05/05
NOMOUTV  : 115 Volts
NOMBATTV :  48.0 Volts
EXTBATTS : 0
FIRMWARE : 83.14.D
END APC  : 2014-11-17 08:25:04 -0500
  • It is very scary log for me. You have 100% BCHARGE, TIMELEFT = 3 min and DSHUTD = 180 sec. That mean that UPS drain the batteries before normal shutdown even begins... Seems that batteries should be replaced and calibrated. – Kondybas Nov 17 '14 at 14:15
  • Thanks for your feedback - this is very useful. It's a brand new APC unit - no battery issues or anything that I know of. It is clear now that I will have to run more diagnostics. 1) Server crashing - no logs to suggest this 2) Server hardware - Dell PowerEdge 510 and APC 750 VA RM 3) OOB on server - DRAC setup, not recording any issues 4) Frequency - Every 3+ days 5) Other resets - Switch is unmanaged so can't get stats. Only server in rack is the Dell 6) Monitoring system remote but corresponds with the local servers uptime – Chris Nov 17 '14 at 14:54
  • @Kondybas Yes, this is a scary log. It's a very bad example using 9 year-old batteries. – ewwhite Nov 17 '14 at 15:03
1

Save your logs to disk. You may change log path to disk by this util or change path manually. At next reboot you can find out a reason.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.