12

Here we have some servers and almost each of them has a dedicated UPS. There are dependencies between them so they must be switched on in the correct sequence. Ultimately we are experiencing serious problems with the power supply, so the servers are shutdown and then restarted in a random order when power is restored. It is not a problem if the servers were switched off during a blackout, it is important they work correctly without any human intervention once power is restored.

Our UPS are quite cheap and the only configuration parameter useful for my goal is power the load xx seconds after power is restored. In theory putting the right delays on each UPS I can fix the order of server restart but I don't trust the UPS will behave as expected.

Is it the right way to go ?
Do high level UPS give other options to fix the restart sequence ?
One final note: my Ups are in the range of 1000 - 2200 VA

Improve this question
3
  • 1
    This is one of the nice things offered by systemd - the ability to define proper dependencies in the startup process. Wait until service X is available before trying to start service Y.
    – MSalters
    Jun 8, 2016 at 9:55
  • 1
    @MSalters AFAIK systemd dependency management only works when the units are handled by the same systemd instance and not for services running on completely different servers...
    – HBruijn
    Jun 8, 2016 at 10:11
  • 1
    @HBruijn: Sort of, network mounts for instance work across servers. That is to say, if server1 mounts a filesystem hosted by server2, then serve1's services which depend on the mount will pause until server2 has started those services. And IIRC you can also have servers wait for DHCP (don't ask me why a server uses DHCP, but it was mentioned in an answer)
    – MSalters
    Jun 8, 2016 at 10:44

4 Answers 4

Reset to default
25

The standard answer for this is "not at all". Fix the software to handle restarts in random order. If you really need SOME servers to start first (example: Active Directory) put them on USV's that are possibly surviving a LOT longer. A low power atom based server is good enough as Active Directory controller and will survive a day on a small USV.

Do high level UPS give other options to fix the restart sequence ?

No. I would say it is generally assumed programmers are competent enough to work around the issue properly.

What you COULD do is:

  • Have servers start "randomly". Except for DHCP / Active Directory there is nothing really demanding an order that can not be fixed.
  • Have a control server after some time (5 minutes) start the services on the various machines in the correct order.

I would say that this type of setup is a lot more common. I would call any software that REQUIRES server starts in a particular order (outside of pure infrastructure) as broken and unfit for business.

Just as note: our own setup is a low cost 20kva USV (low cost because we got one used) for the servers, with a slaved 2000VA USV for a machine serving as "root" of the network (and backup machine). Slaved means that the USV is behind the big one - so it only switches to battery when the large one (that lasts between half an hour and 8 hours depending on how much of our computing grid is online) is going into terminal shutdown.

Improve this answer
12
  • 2
    I think this is sometimes easier said than done (AD, as you say, is an obvious example) but I do agree. The correct solution is to work on eliminating dependencies for things like start order of servers or services. If nothing else, it should be possible on a web app, for example, to write code that says "If I can't connect to my back-end, 'sleep' and try again later rather than crash horribly".
    – Rob Moir
    Jun 8, 2016 at 8:26
  • he problem with AD is not even AD - it is mostly IPv4 DHCP which is not prepared for computers being online before the dhcp server. Ipv6 handles this ;)
    – TomTom
    Jun 8, 2016 at 9:17
  • That's true. IPv4 is a pain... and I still get people here asking me why we need to bother with "this newfangled IPv6 rubbish".
    – Rob Moir
    Jun 8, 2016 at 9:53
  • 1
    "It is generally assumed programmers are competent enough to work around the issue" - you must not do much programming! No, in all seriousness, there's a huge amount of reasons a system may need to be brought up in a specific order. Yes, software should "fail gracefully" and re-try connections that are broken, but it's not always possible. From what I recall, some of the nice PDU's have the ability to start/stop individual ports, so perhaps something could be done there.
    – SnakeDoc
    Jun 8, 2016 at 15:14
  • 1
    I had to search for "USV" and found "Unmanned surface vehicle". I know this is wrong, but I want it to be right.
    – Braiam
    Jun 8, 2016 at 21:02
14

Managed Power Distributions Units (rather than the UPS) often do support customised delays in enabling individual outlets after power is resumed.

Typically that is to prevent circuit breakers from tripping when a cabinet full of systems powers up at the same time immediately after power is restored, but that can also be used to preserve the boot order of your system dependancies.

Improve this answer
1
  • Yeah, correct. This is advanced functionality and it is not assumed the USV is actually connected to the servers - but it is powering racks that then use PDU's to handle the details.
    – TomTom
    Jun 8, 2016 at 9:19
6

I had this exact issue. The only difference being we invested in sturdy rack mounted APC power units (for example APC SmartUPS 3000 ). With the APC PowerChute network shutdown software (PowerChute Network Shutdown software) , I'm able to shut down and bring up servers in a specific order. Another handy feature of the software was setting the servers to shut down at the very last minute, i.e. calculating how much battery power the APC units had left and shutting down the servers with just enough time for them properly shut down instead of just powering off.

The software is...not user friendly but it's nothing difficult if you take some time to figure it out. If you're interested in investing more in your infrastructure, this is definitely the route to go.

Improve this answer
2
  • 1
    We have Apc Smart Ups too, some of them are relatively old and perhaps have weak batteries. It is difficult to make some tests on them because they are in production. Besides we lack any freak load, I mean a load which can bear a sudden power down without problems. That said every time I simulated a blackout the UPS behaved differently that expected, it could be due to a misconfiguration but my feeling is that those UPS are not very reliable.
    – Filippo
    Jun 9, 2016 at 8:51
  • @Filippo certainly YMMV but I have a mix of SmartUPS 3000 and 3000XLMs across multiple sites using the PowerChute software for going on 3 years and after figuring out the software, there is certainly a learning curve and some testing is needed, it's been pretty solid.
    – Winski Tech
    Jun 9, 2016 at 17:53
2

It sounds like the UPS units are low-cost and not capable of being configured for a specific output-on wait time after power is restored (some higher end units are). To get the same functionality, you need to pick a specific host to always power on right away (maybe whichever system is allowed to boot at any time) and leave all the other servers in a powered off state (configured in bios to return to power off when AC is applied, and to honor the Wake On Lan magic packet to power on when told to do so). Then, on the main host that does boot, run a script/utility to time the transmission of the WOL magic packet to each host.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.