Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am using func to perform parallel commands on our servers.

The other day, we had an issue when a service restart of puppet via func made all our severs hit our puppetmaster at the same time.

My question: How can I execute the same exact command on a set of servers while adding a delay before it's executed on the individual servers?

E.g.: random_delay && service puppet restart

I am interested in the random_delay part of the command.

share|improve this question

2 Answers 2

up vote 5 down vote accepted

sleep $((RANDOM % MAXWAIT)) where MAXWAIT is the maximum desired delay in seconds.

share|improve this answer
    
Seems to get stuck on "1" a lot if you echo that out... Thoughts? –  Corey S. Jan 7 '12 at 3:00
    
Noticed this as well. Try this, for x in 1 2 3 4 5; do RANDOM=$x; i=$((RANDOM%10)); echo $i; sleep $i; done Obviously, MAXWAIT in S19N's answer is similar in effect to 10 in my answer. Change the RANDOM seed and add puppet code to whatever you desire of course. –  user1125087 Jan 7 '12 at 3:11
2  
@CoreyS. $RANDOM is seeding off the last value, hence the looping (I've gotten stuck on 5/6/7, 2/7, and 5). sleep $((RANDOM % MAXWAIT)) is the correct way. I'll edit the answer as such. –  Jeff Ferland Jan 7 '12 at 3:11
    
This is called splay in Pro Puppet. It is used to prevent a thundering herd, where many machines are doing the same thing simultaneously. In addition to this, I had success with cron scheduling using fqdn_rand. –  François Beausoleil Jan 17 '12 at 13:46

I quite like S19N's innovative, yet less than ideal solution. I only say it's less than ideal, as it's still largely non-deterministic when stuff will actually happen. I'd much rather be able to guarantee when stuff will happen, and what stuff will happen when it does.

Puppet orchestration is actually a hard problem.
One of the "best-practice" solutions is to use MCollective which will not only allow you to configure when puppet runs on your cluster of machines, but you can also use it for other similar orchestration problems.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.