Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am shipping a bunch of ESXi 5.1 servers to remote offices where they will be powered via APC UPS.

I would like to have the UPS trigger a shutdown of the connected server - I would then rely on the ESXi configuration to take care of the shutdown/suspension of the VMs hosted on it.

I can see that APC have a solution documented using their PowerChute Network Shutdown, but this involves setting up an extra server per office, and requires network cards on each UPS. We are generally using UPS without a network card (e.g. Back-UPS Pro) - they come with a USB connector, and they are readily available in the locations where our offices are.

How can I connect a UPS to a ESXi host via USB, then have ESXi detect a power outage and then act accordingly? Has anyone managed to do this.

share|improve this question
1  
Have you timed the shutdown process of the VM's through the host shutdown? Can the battery hold long enough for that period? –  ewwhite Jan 4 '13 at 14:38
    
Thanks for pointing this out. Not yet - at this stage I am just shipping the ESXi servers for running a Domain Controller, but I am sure that once we have the resource in place we will add a few more servers, at which point the timing might change. –  dunxd Jan 4 '13 at 14:54
    
The shutdown policy is quite long by default. But to be honest, I don't run UPS shutdown on my ESXi hosts or clusters. Seems counter-intuitive, but has never been an issue. –  ewwhite Jan 4 '13 at 15:00
    
Why bother having UPS on your ESXi hosts at all then? If the power goes because of an outage or because the battery ran down you get the same result. –  dunxd Jan 4 '13 at 15:21
    
To weather brief outages. But at my bigger sites, I have 2-4 hours of UPS power available for the VMWare cluster, storage and networking. –  ewwhite Jan 4 '13 at 15:22
show 3 more comments

7 Answers 7

up vote 4 down vote accepted

According to APC, this is not possible and you require Powerchute Network shutdown. We tried this a number of times with USB and found no solution.

VMWare has info here on using the APC approved solution.

Would also think SmartUPS would be a better choice and you can fit with network card. Naturally more money but if your servers are at all important, that cost should be worth it. Alos gives you more monitoring and alerting which might be useful at a remote site. You also need to assure sufficient runtime for all VMs to cleanly shutdown and then shutdown the host

share|improve this answer
1  
This seems like the most sensible answer being supported by both vendors. Unfortunate that VMware haven't thought to build anything into ESX/ESXi that does this natively. The Network solution requires that at least one network switch is powered via UPS too. –  dunxd Jan 4 '13 at 15:34
1  
It would not make much sense to not power network switches via UPS... they consume very little corrent and are critical to any network operation. –  Massimo Jan 4 '13 at 16:19
add comment

Yes, it's possible. Here are details of my similar setup.

Hardware configuration: APC Smart-UPS 1500 connected to the ESXi 5.1 Host via USB. A Linux virtual machine running on this ESXi host. UPS is connected to this VM using ESXi USB pass through option.

Software configuration: NUT (Network UPS Tools) master running in the VM, and native ESXi NUT slave running on the ESXi host.

Shutdown logic: VM is running the UPS driver usbhid-ups which is responsible for the communication with UPS via USB. The upsd process connects to the UPS through the usbhid-ups driver and monitors the UPS state. The upsmon master process running on the same machine connects to the upsd and initiates the shutdown. ESXi host is running the 2nd instance of upsmon which also connects to the same VM upsd via internal network.

On power failure the following sequence takes place:

  1. UPS via usbhid-ups reports to upsd about power failure.
  2. (optional, useful if you want to shutdown in few minutes instead of Low Battery) upsmon on the VM initiates upssched 5 minutes timer. Timer is aborted if power is restored.
  3. When timer fires or when UPS reports Low Battery, the upsmon raises the FSD (forced shutdown) flag to upsd.
  4. In a stand-alone NUT configuration the FSD flag would shutdown the machine. But here the shutdown command is replaced by simple logging like "I should shutdown now but I am waiting for the host instead". And does nothing.
  5. The FSD flag is also read by ESXi upsmon, which initiates the ESXi host shutdown.
  6. ESXi host shuts down all virtual machines one by one. The important thing is that VM which runs the upsd should be shutdown last (using ESXi startup/shutdown sequence configuration).
  7. Important: this VM must have vmware tools installed. When it receives the guest shutdown command from the host, the vmware-tools shutdown script is being started. This script checks for the /etc/killpower flag. If no flag, it does nothing (this means user activated linux shutdown, not the UPS event). But if the flag exists (FSD active), then this script sends to UPS the delayed powerdown command (say, in 3 minutes).
  8. After running vmware-tools script the guest VM shuts down.
  9. ESXi sees the last VM poweroff state and goes down itself (it takes around 1 minute because there is no other machines running now).
  10. In 2 remaining minutes the UPS cuts off the power.
  11. When power is restored, the ESXi starts and powers on all VMs. The UPS monitoring machine must be started first (the same configuration as for shutdown order).

Downloads:

The NUT for Linux could be installed from package.

The native NUT client for ESXi server can be downloaded using last link on this page: http://www.networkupstools.org/download.html

Some my scripts and conf files are here (only changed lines are shown): http://pastebin.com/KkEeanK1

Notes:

Of course there are more details, and it took some time for me to make this working as it should. But now it performs very nicely. This system accounts for the cases when you just shutdown the monitoring VM from inside (vmware-tools script is not run), or if it's a ESXi host initiated VM shutdown (no /etc/killpower flag, so no UPS load off), or if it's a ESXi shutdown (the same). The only important is to have this VM running ASAP after host boot, and shutdown it last (so host down time is predictable - as said above, it is around 1 minute for me and 2 more minutes I reserve just in case).

My UPS monitoring Linux VM is also Samba/NFS sharing server for backup storage, the NAT/DHCP server for VMs, and some other light-weight services. It takes around 22MHz of ESXi CPU shares and around 10MB of active RAM when idle. Due to using the NUT you can power more devices from the same UPS if required, and all they can be shut down gracefully. No PowerChute and/or expensive Network Monitor Card is required.

share|improve this answer
add comment

You might consider using the USB device passthrough functionality to a guest running PowerChute or other software able to monitor the UPS's health and capable of triggering a shutdown on the ESXi host (e.g. apcupsd). ESXi officially only supports a very limited number of USB devices for passthrough, but people have been attaching and passing through different classes of devices for a while already with varying success, but the APC UPS USB seems to work according to this walkthrough for a Windows VM or this one for a CentOS Linux VM.

share|improve this answer
add comment

Have a look at vSphere Management Assistant (vMA) from here We use it at my office for doing what you are attempting, however with Smart-UPS connected via USB rather than Back-UPS.

share|improve this answer
    
Please add more detail as this is an undocumented setup as far as APC or vmware are concerned. –  dunxd Jan 7 '13 at 10:24
add comment

While possible (probably/generally), I don't think an automated shutdown of a computer on battery power is a good idea. If you're going to do that, then for most practical intent and purposes, you should probably just save yourself the money of a battery-backed UPS, and let the loss of power shut down your machine for you. (Granted, a clean shut down is always preferable to a power loss, but you seem to be missing out on the point of having a battery time of longer than a couple minutes if you automatically shut every thing down when you lose the power feed.)

The way I've always handled it is to have monitoring alert the SAs when the power goes down, so the SAs can use their grey-matter to decide when (or even if) to shut down the servers. If it's a brief outage, it may not be a good idea to shutdown the servers at all, or you may want to leave some servers up and running as long as possible, and only shut them down before the battery's about to die. Really seems to me like a decision-making task better suited for a human than a simple rule.

share|improve this answer
    
You don't have to configure your UPS to trigger a shutdown immediately, but you do want it to shut off before the batteries drain completely else you will have to buy more batteries, particularly in some of the locations where I work and the power goes daily. It is great to get human's involved of course, but you don't always have a System Administrator in a remote office. –  dunxd Jan 4 '13 at 15:39
    
@dunxd Good point - I'm more accustomed to HA environments where at least some of the servers have to stay up, come hell or high water, so the name of the game is figuring out how to best ration to the power (selectively shutting down devices) to create the least service impact possible, which won't be everyone's focus or use-case. –  HopelessN00b Jan 4 '13 at 16:05
add comment

In the olden days of baremetal installations, APC PowerChute Plus was an essential part of my install process. Using the simple serial signaling cable and their Red Hat-only binary, it was easy to setup rules to govern a locally-attached server. Basic email notifications for UPC battery events, line power events and shutdown actions were available:

POWERCHUTE MAIL MESSAGE
Message from PowerChute@Bonanza:

UPS on battery: Blackout 000.0 V. 

and

POWERCHUTE MAIL MESSAGE
Message from PowerChute@Bonanza:

Normal power restored: UPS on line.  

or

POWERCHUTE MAIL MESSAGE
Message from PowerChute@Bonanza:

Shutdown started.  

Plus a reasonable interface to see what was happening...

enter image description here

That software eventually went commercial (or was buried on the APC website). There were a few open-source approaches to provide something similar. But this all gets complicated with single VMWare ESXi hosts.

It seems like this is something that VMWare should have incorporated into the base hypervisor. It's basic and could offer a decent level of protection for users. The most common remedies I see now are USB passthrough to a dedicated VM, a network daemon approach or doing what I do; not configuring any automatic or battery shutdown...

Granted, I typically go with a UPS that can support the system load for an hour or more, but extended outages DO happen. Maybe an alternative is to collect a few low-cost or refurbished network interface cards and plan to buy SmartUPS devices as a minimum...

share|improve this answer
add comment

Check out the following link. Not the most elegant solution, but a very practical, very straightforward solution. There are possible drawbacks in terms of security (depending upon your particular network design, the guests loaded on the Hosts, and the access users have to those guests but you get to make that call.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.