5

I am working on a project to reboot a great deal of computers. One of the important requirements is to stage the reboots so all of the machines aren't rebooting at once (too fast and it will cause issues with the SAN).

I tried doing it in a workflow by throttling to 50 parallel actions and adding a 15 second delay (200 reboots per minute).

workflow Bounce-Computer {
param(
[string[]]$Computers
)
foreach -parallel -throttlelimit 50 ($computer in $Computers) {
    Restart-Computer -PSComputerName $computer -Force
    Start-Sleep -Seconds 15
    }
}

But I ran into an issue where the workflow would hang if WMI was broken on a target computer.

Outside of fixing the WMI on all of the target machines (there are several thousand), how would I go about doing something like this in a controlled way? Jobs?

2
  • What's the desired result for machines that get an error? Just ignore them? Also, your throttlelimit is a bit high. Is the machine this script is executing on pretty beefy? Try with a throttlelimit more like 5.
    – Colyn1337
    Apr 9, 2015 at 20:33
  • Ignoring the machines in error is fine as I run a script later to determine the last startup time (to verify the reboot) and I have error logging there to deal with that. The machine that I am running it on is a VM that has 16 vCPU and 32 GB of memory. I have run with these throttle limit levels before and I have had no issues.
    – Acerbity
    Apr 9, 2015 at 20:36

4 Answers 4

0

I think there's more than one way to tackle your problem, at least from what I see as the cause. Here's how I'd do it:

Workflow Invoke-MassRestart{
    Param
    (
        [parameter(mandatory=$true,
                   ValueFromPipelineByPropertyName=$true)]
        [string[]]
        $ComputerName,

        [int]
        $Throttle = 5,

        [int]
        $Delay = 5
    )

    Foreach -parallel -ThrottleLimit $Throttle ($Computer in $ComputerName){
        Sequence {
            InlineScript{
                [void](Restart-Computer -PSComputerName $using:Computer)
            }
            Start-Sleep -Seconds $Delay
        }
    }
}

Let me know if there's any parts of this you want me to explain. I'd like to call out my casting of Restart-Computer to the [void]. That basically tells the system to issue the command and move on. I think you were getting hosed when it was waiting for 50 operations to report a status of some kind. Also note the unique scoping needed inside the InlineScript{} block. I also decided to make it so that it could accept pipeline input from Get-ADComputer.

1
  • Hey, thanks a ton. There are a couple of things that I don't understand, but I need to leave for the day. I will take a look at it tomorrow and comment if I'm still lost.
    – Acerbity
    Apr 9, 2015 at 21:16
0

Without reworking your script too much you could let the script catch the WMI timeout error (if it throws a catch-able one during the hang) and then reboot the next computer. The catch statement could output the problem computer and error for you to review later.

Workflow Bounce-Computer {
 param([string[]]$Computers)
 foreach -parallel -throttlelimit 50 ($computer in $Computers) {
    try{
        Restart-Computer -PSComputerName $computer -Force -ErrorAction stop
        Start-Sleep -Seconds 15
    } 
    catch [AppropriateWMIExceptionError]
    { 
       echo 'WMI timeout error' #Or another action to note the problem computer
    }
}
2
  • Thanks for responding. The problem is that it never generates an error. It hangs for several minutes at least (if not indefinitely).
    – Acerbity
    Apr 13, 2015 at 18:03
  • Hmm. If you're using Powershell 3.0 the -Timeout and -Wait parameters for Restart-Computer might be useful in that case. I couldn't simulate a WMI hang to test it but you could I suppose.
    – chong
    Apr 13, 2015 at 19:53
0

I would use the job engine instead and do something like below. This is not fully tested, but it should work might have missed something somewhere.

This way you also get them to run in parallel and if one restart hangs it is only that job taking time and if a job takes too long time we remove this job.

Code below:

function restart-myComputer (
[Parameter(ParameterSetName="restart", Mandatory=".")]
[Parameter(ParameterSetName="bg", Mandatory=".")]
[string]$computer,
[Parameter(ParameterSetName="restart")]
[int]$wait,
[Parameter(ParameterSetName="bg")]
[switch]$bg)
{ 
  if ($bg) {
    Start-Job -name $computer -ScriptBlock ([scriptblock]::create("Restart-Computer -PSComputerName $computer -Force -ErrorAction stop")) > $null
  } else {
    Restart-Computer -PSComputerName $computer -Force -ErrorAction stop
    Start-Sleep -Seconds $wait
  }
}

function remove-myJobs (
[Parameter(ParameterSetName="remove")]
[string[]]$names,
[Parameter(ParameterSetName="remove")]
[int]$maxRunTime = 30
{
  $d = get-date
  [string[]]$stillRunningJobArr = @()
  foreach ($name in $names) {
    $job = get-job -name $name 
    $diffTime = $d - $job.PSBeginTime
    if($diffTime.TotalSeconds -gt $maxRunTime) {
      remove-job -name $name -Force
      write-host "Removed job $name that ran longer then $maxRunTime seconds"
    } else {
      write-host "Job $name has still not been running for more then $maxRunTime"
      $stillRunningJobArr += $name
    }
  }
  return $stillRunningJobArr
}

function remove-allMyJobs (
[Parameter(ParameterSetName="remove")]
[int]$maxRunTime = 30)
{
  $d = get-date
  foreach ($job in get-job) {
    $diffTime = $d - $job.PSBeginTime
    $name = $job.name

    if($diffTime.TotalSeconds -gt $maxRunTime) {
      remove-job -name $name -Force
      write-host "Removed job $name that ran longer then $maxRunTime seconds"
    } else {
      write-host "Job $name has still not been running for more then $maxRunTime"
    }
  }
}

function restart-myComputers (
[Parameter(ParameterSetName="restart", Mandatory=".")]
[Parameter(ParameterSetName="bg", Mandatory=".")]
[string[]]$computers,
[Parameter(ParameterSetName="bg", Mandatory=".")]
[int]$maxConcurrentJobs,
[Parameter(ParameterSetName="bg", Mandatory=".")]
[Parameter(ParameterSetName="restart")]
[int]$wait,
[Parameter(ParameterSetName="bg")]
[switch]$bg)
{
  [string[]]$restartedComputerArr = @()
  if ($bg) {
    foreach ($computer in $computers) {
      if((get-job -state 'Running').Count -gt $maxConcurrentJobs) {
        $restartedComputerArr = stop-myJobs -$restartedComputerArr -maxRunTime $maxRunTime
        sleep $wait # Wait to get as many jobs to complete as possible.
      }
      Restart-myComputer -computer $computer -bg
      $restartedComputerArr += $computer
    }
  } else {
    Restart-myComputer -computer $computer -wait $wait
  }
  remove-allMyJobs
}
0

I ended up rewriting the script to use the VIC. WMI in this environment was just too unstable to use.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .