15

I am using func to perform parallel commands on our servers.

The other day, we had an issue when a service restart of puppet via func made all our severs hit our puppetmaster at the same time.

My question: How can I execute the same exact command on a set of servers while adding a delay before it's executed on the individual servers?

E.g.: random_delay && service puppet restart

I am interested in the random_delay part of the command.

30

sleep $((RANDOM % MAXWAIT)) where MAXWAIT is the maximum desired delay in seconds.

5
  • Seems to get stuck on "1" a lot if you echo that out... Thoughts? – Corey S. Jan 7 '12 at 3:00
  • Noticed this as well. Try this, for x in 1 2 3 4 5; do RANDOM=$x; i=$((RANDOM%10)); echo $i; sleep $i; done Obviously, MAXWAIT in S19N's answer is similar in effect to 10 in my answer. Change the RANDOM seed and add puppet code to whatever you desire of course. – dtbnguyen Jan 7 '12 at 3:11
  • 2
    @CoreyS. $RANDOM is seeding off the last value, hence the looping (I've gotten stuck on 5/6/7, 2/7, and 5). sleep $((RANDOM % MAXWAIT)) is the correct way. I'll edit the answer as such. – Jeff Ferland Jan 7 '12 at 3:11
  • This is called splay in Pro Puppet. It is used to prevent a thundering herd, where many machines are doing the same thing simultaneously. In addition to this, I had success with cron scheduling using fqdn_rand. – François Beausoleil Jan 17 '12 at 13:46
  • 1
    This seems to depend on bash-specific extensions. In dash echo sleep $((RANDOM % 900)) returns sleep 0. This means it is not safe to use in #!/bin/sh scripts or places like crontab. (The problems seems to be that dash does not understand $RANDOM) – Gert van den Berg Nov 22 '18 at 12:28
1

I quite like S19N's innovative, yet less than ideal solution. I only say it's less than ideal, as it's still largely non-deterministic when stuff will actually happen. I'd much rather be able to guarantee when stuff will happen, and what stuff will happen when it does.

Puppet orchestration is actually a hard problem.
One of the "best-practice" solutions is to use MCollective which will not only allow you to configure when puppet runs on your cluster of machines, but you can also use it for other similar orchestration problems.

0
0 */12 * * * root perl -e 'sleep int(rand(43200))' && service puppet restart

This will sleep a random amount between 0 seconds and 12 hours (43200 = 12 x 60 x 60) every 12 hours

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.