3

Problem: The SSDs in our servers require a power cycle (i.e. off/on, not reset/warm reboot) after a firmware update.

Thoughts: Using 'ipmitool chassis power cycle' I can cycle the server's power. However this would cut the power while the system is still running, filesystems are mounted, etc.

What I basically want is a delayed power cycle so the system has a chance to halt. But I guess that would have to be implemented on the server's IPMI board, so it's not really an option.

My initial idea was to dynamically create a ramdisk containing the tool and libs and somehow integrate that into the halt process. I saw there's a /etc/init.d/halt, so that would be my starting point. Although I believe the kernel at some point in the shutdown process starts to kill off remaining processes. So I'm not even sure if that's a viable way.

Question: What would be the best way to execute ipmitool (or any other command), after the system has halted and all regular filesystems are unmounted?

7
  • You're doing firmware updates on a live server, while it's actually serving? Pretty daring of you. Oct 10, 2012 at 15:07
  • The SSDs that I'm updating are part of a zpool mirror. Half of the mirror is talken offline for the update. After the power cycle the firmware upgrade process waits for the pool to be resilvered, then takes the other half offline, updates and power cycles again. Since the FS the SSDs use is ZFS and the OS itself resides on an ext4 HDD I was already contemplating just calling 'sync' and then cycling the power since nothing should break. But I'm still interested if there's a proper way to execute a command after system halt. Oct 10, 2012 at 15:20
  • @MichaelHampton Well, as they say, fortune favors the bold. Computer systems might not, but I hear that fortune does. Oct 10, 2012 at 15:22
  • Using your DRAC card? Oct 10, 2012 at 15:23
  • 1
    Perhaps I'm missing the blindingly obvious, but shouldn't IPMI access be happening on a separate LAN that is accessed from the management console and ideally inaccessible to the host? Oct 10, 2012 at 18:17

4 Answers 4

7

I would take it down to single user mode, unmount any non-OS filesystems and remount any existing ones as readonly. Then run your firmware update tool and 'ipmitool chassis power cycle'.

It's not pretty or clever but should be safe enough

1
  • Very straight forward and it solves the problem brilliantly! I implemented the updating and power cycling as a rc script that runs one time when single user mode is entered, since init sends SIGTERM and 5s later SIGKILL to all processes in it's process group upon changing runlevel. Also had to restart the ipmi driver to have /dev/ipmi0 on which ipmitool operates. Overall this works great! Oct 10, 2012 at 16:52
2

Do a normal full shutdown and power-off; count to ten and then power the machine back on with wake-on-lan or IMPI to the BMC (DRAC in the case of Dell?). If you want to keep it entirely local to the machine, some BIOSes will let you specify a wake-up/power-on time via ACPI - see http://www.mythtv.org/wiki/ACPI_Wakeup . E.g. set power-up time to (now) + (normal shutdown duration) + (a small safety margin), then run shutdown.

I've no idea if that's actually possible with server-class hardware - please let us know if you try it.

1
  • 1
    This works too and I would have done it that way if Fuzzyfelt hadn't come up with his single user mode solution which I feel is a little more elegant since it doesn't depend on the clock and the shutdown to finish in n-minutes. However using /sys/class/rtc/rtc0/wakealarm I was able to have the machine turn back on after the shutdown. Nice solution! Oct 10, 2012 at 16:56
1

You're looking for a watchdog. A timer that is activated by your software and resets the computer if the timer expires without being refreshed. For this you either need a BIOS with a supporting timer to turn the machine on, an external IPMI manager, or the good fortune for the IPMI power cycle support to work on your machine directly.

As a starting point, check out IPMI util which also includes the tool with a "power cycle" reset.

If you can set the watchdog timer active, then establish a runlevel that starts the timer with a timeout function of hardware poweroff. The last effect of your shutdown-without-ACPI-off runlevel should be to kill the timer update process.

0

At startup (instead of shutdown).

Using a network boot.

You need to power cycle the physical drives, so I suggest you reboot the machine, telling it to boot from the network, once done, you power cycle the drives, and then reboot from the drives.

2
  • That was another idea I had. However I'm trying to keep the whole process contained to the machine that I'm updating. If I'm involving other machines (e.g. telling the PXE server to provide me with a different boot image on next reboot) I might use mcollective (which we got in place anyways) to tell a management machine to execute the power cycle on me once I'm unreachable (i.e. system halt completes). It's definitely an option if I don't find a way to do it local. Oct 10, 2012 at 15:26
  • Another idea I had was to use GRUB's once-only option. I'd just like to avoid this because, like with the PXE boot, it would mean going through the BIOS's RAM test four times instead of two. With 256GB RAM in the servers that makes a difference of almost 20 minutes of downtime (a full reboot takes 11 minutes of which a little over 9 go to the bios mem test). Oct 10, 2012 at 15:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .