Background: I manage two sites in a town with overhead powerlines, winter storms, people with guns, and the occasional lightning strikes. These circumstances combine to create some rather hostile seasonal power events. We typically lose a small but significant number of devices to mains power events each year. The devices on online UPSes seem to be fine. It is the devices connected directly to mains power that seem to be vulnerable. One of our sites happens to be in the same building as a master electrician whose company also happens to be, among other things, the power company for some remote villages. So, for better or worse, this solution is mainly informed by an electrician rather than an IT professional.
There seems to be three common threats to devices directly connected to mains power:
- Undervoltage or Brownout
- Harmonics and transients
- High Voltage (as in several thousand volts)
Undervoltage or Brownout
In my area there are two common causes of undervoltage or brownout:
- a large dynamic load in the neighborhood
- loss of a phase
A large dynamic load in your neighborhood can cause the voltage feeding your site to fluctuate. The voltage you see at your site will be maximum when the load is switched off, and minimum when that load is switched on. The dynamic load in the neighborhood of one of our sites (the electrician thinks it might be a deep fryer or water heater) causes the voltage on one of the two phases to fluctuate between 106 and 142 volts. Occasionally and momentarily, that same phase inexplicably drops to 98 volts.
When only one phase of a multi-phase service is interrupted, the interrupted phase's voltage does not go to zero. This is because many multi-phase devices (like furnaces, water heaters, motors) will backfeed some voltage back to the interrupted phase. So, other single-phase devices directly connected to mains power (like, say, access switches and monitors) will see some low AC voltage. I measured between 20 and 80 volts during a recent phase interruption.
Based on my experience so far, the 106-142 volts doesn't seem to cause harm as long as the voltage is stable. However, the 20 to 80 volts experienced during that recent phase interruption proved harmful. During that event, an unmanaged switch and the power supplies of two monitors were damaged and had to be replaced. This is roughly consistent with the 90-240 volt input power specifications of many universal power supplies. In other words, when one phase is interrupted, you are probably subjecting all of the devices directly connected to that circuit to power that is out-of-specification.
Harmonics and Transients
Some industrial devices (think big motors, servos, heaters) backfeed harmonics and/or transients back onto the power grid. Once on the power grid, the whole neighborhood's power can be polluted by these transients and harmonics. Europe requires that some devices be immune to mains power harmonics and transients. North America has no such requirement. In a previous career I designed electronic devices that underwent immunity testing for Europe. When subjected to mains power transient and harmonics, the output voltage of power supplies occasionally went wildly of specification. Those problematic power supplies I tested are blackboxes indistinguishable from the handful I see powering switches, routers, monitors, and laptops at our sites.
The moral of the story is this: If you have enough transients and/or harmonics on your mains power, it is certainly possible that some of your adapters will occasionally produce an output voltage that is out-of-specification.
This is the fun one. By high voltage, I mean thousands of volts. There are two common reasons you might see high voltage at your site:
- short-circuiting of primary and secondary transformer windings
- lightning strike
In our area, the voltage of the conductors at the top of power poles is commonly 14,400 volts. The grey cylindrical transformers you see on poles and those ground level metal boxes you see around subdivisions transform that power down to the hundreds of volts that is typically fed to your site. Apparently people like to use the pole-top transformers for target practice. The problem with that is that the transformers are oil-cooled. When the oil-bath is penetrated by, say, a bullet, the cooling oil leaks out. Without oil, the windings eventually heat up enough that the insulation between the primary and secondary windings break down. When that happens, the full 14,400 volts or so is applied to your site until one breaker or another trips. This all usually happens faster than the eye can see, but normal circuit breakers, fuses, and surge protectors don't handle this event correctly or quickly enough to prevent damage to your devices.
In our area, at least, lightning strikes that cause problems with electrical equipment are extremely rare so I don't have any experience with them. As best I can tell, however, the measure for protecting equipment are similar for both shorting transformers and lightning strikes.
Power Companies' Tolerances
You might think that the threats I enumerated above are somehow the responsibility of the power company to mitigate. As far as I have been able to establish, in my area of BC at least, it is not. BC Hydro's "System Voltage Control" policy, for example, explicitly permits voltage deviations of +/-20% for minutes at a time. I have yet to even find a policy that contemplates transients and harmonics. And, of course, the power company is not likely to be responsible or liable for vandalism-by-bullet of their transformers.
First of all, online UPSes seem to offer the best protection against all of these threats. They also support graceful shutdown of equipment. On my sites, at least, it is impractical both physically and cost-wise to have all vulnerable devices on an entire site powered by online UPSes. So, what to do about the devices that are necessarily connected to mains power? I am using two separate strategies to deal with the threats:
- power monitoring and contactor site disconnect
- HRC fuse and surge protection device
Power Monitoring and Contactor Site Disconnect
The concept here is that your entire site is connected to mains power if, and only if, the mains power meets certain power quality criteria. When those criteria are not met, power to the entire site is disconnected. Set up correctly, this will eliminate completely the damage caused by undervoltage and brownouts -- if one phase is interrupted, both phases are immediately disconnected completely from mains power.
For reference, I'm using the Bender VME420 along with a contactor in line with the panel feeder to protect my most problematic site. You can program it to fit your needs. It basically just opens an electromagnetic contactor when the power goes out of my specifications -- which it occasionally does. I've got it set to re-close after mains power has returned to normal for a while.
I'd like to also monitor for harmonics and transients, but I haven't found a device that does that yet.
HRC Fuse and Surge Protection Device
The strategy here is that when high voltage is applied to your feeder, two devices work in tandem. First, a purpose-built Surge Protection Device (SPD) acts as an electrical shunt and crowbar to limit the voltage reaching your site to what is typically hundreds of volts. It does this by passing very high current very soon after the appearance of the high voltage. By itself an SPD isn't very useful. It will quickly overheat and fail open thereafter providing no further protection.
The other parts of the puzzle are high-rupture capacity (HRC) fuses upstream of the SPD. When the SPD shunts the high current required to keep the voltage at a safe value, the fuse upstream of the SPD will blow. This has to happen fast enough that the SPD does not overheat and fail. The high rupture capacity of the fuses is required because the voltages we are dealing with are high enough to arc across plain-old fuses even after they have blown.
For reference, I'm using this Siemens SPD at my most problematic site.
This strategy should protect your site against both lightning strikes and transformer shorts.
You'll Need an Electrician
Installing any of these devices requires a licensed electrical contractor in my jurisdiction and most other jurisdictions as well.
Budget and Business Case
All told, the parts and labor to install this protection was about $2500 CAD. I estimated that brownouts were costing $1000 to $2000 per year in lost productivity and parts. So I expect a payback in two or three years. If our transformer shorts or we have a lightning strike, it should pay for itself immediately. On the other hand, if you have rather clean and reliable mains power, all of this stuff is probably not worth it.