Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have the responsibility for changing the local administrator account password on my environment of 16,000 servers; I wrote a PowerShell script, but it took too long, so I added multi-threading using the powershell runspace factory to break out the 16,000 into manageable pieces.

There's a ~5% error rate (~800 servers); of these, 75-100 are clear errors that can be troubleshot (username not found, access denied, etc.), and 700-725 get the error message "The network path was not found".

However, pinging the servers gets a response and server engineers tell me they are operational, I have access, and that both PowerShell and WMI are running and functioning.

I have no idea where to begin troubleshooting. Here's the logic and code I'm using:

I use FQDNs, however my company tends to have servers listed differently in DNS to their FQDN, and the 2 choices will not resolve to one another. so servera.production.active.directory will not resolve to servera.mycompany.com. This function determines a valid FQDN for connecting and setting the password, returning either a valid FQDN or an empty string:

function get-validfqdn([string]$server, [string]$domain){

    $fqdn = $server + "." + (get_FQDN $domain)
    $altdn = $server + ".mycompany.com"

    if(Test-Connection -count 1 -computer $fqdn -quiet -TimeToLive 80){
        $valid = $fqdn
    }
    elseif{
        $valid = $altdn
    }else{
        $valid = ""

    return $valid
}

I attempt to execute the password change using the following code, embedded in a module and performed for each server in the list we're processing (this is a long function due to the PowerShell runspace factory code).

function Set-ServerPass([string]$filepath){

    $servers = Import-CSV $filepath
    $results = @()

    foreach($server in $servers){

        $svr = $server.Server
        $password = $server.Password
        $domain = $server.domain
        $fqdn = get-validfqdn $svr $domain

        if ($fqdn -ne ""){
            Try{
                $admin = [adsi]("WinNT://$fqdn/Administrator, user")
                $admin.psbase.invoke("SetPassword", "$password")

                $result.Error_Code = "0"
                $result.Error_Msg = "The operation was sucessful"

            }Catch{
                $error_msg = Trim_ExceptionMessage $_.exception.Message

                $result.Error_Code = "1"
                $result.Error_Msg = $error_msg
                $results += $result
            }
        }else{

            $result.Error_Code = "51"
            $result.Error_Msg = "The remote computer is not available"
        }
    } 
    return $results
}

Notes: Test-Connection filters out servers that would otherwise be unavailable; a timeout on this function defaults to ~180 seconds (3 mins x 1600 servers = too long).

This code works on 95% of the servers, and reports accurately after a year of running this script. However, server engineers are starting question whether this script works because when I report problems they're not seeing how or why I would get "the network path was not found" error when all of their tests say it is working fine.

Troubleshooting steps so far:

  • Run on different computer
  • Run as different admin
  • Run at different times of day - this was to prevent possible server activity from interrupting the script (application patching, reboots, etc)

This last 2 months I have manually troubleshot each of the 800 servers and ran the script some 15 times on just the failed servers. Re-running nets me about 10-300 more passwords being reset, but doesn't catch all of them, and it is very inconsistent.

On 3 occasions the server engineers reported no problems, I re-ran the script and it reset all of them with no errors.

So my questions are: what could be causing the error, and what should I look at to determine the the root cause? Settings on server? Settings on my workstation?

Setup is as follows: Windows XP Pro SP3. Servers are Windows Server 2003 or Windows Server 2008 R2. These errors occur on both server operating systems.

share|improve this question
    
yikes that sounds painful, we have more devices then this and we just disable the administrator account! Truly not for everyone but once you have the process down for recovery its not that bad. Other then that this is quite an interesting problem. In this case I suspect DNS for some reason. –  tony roth May 3 '13 at 20:42
    
Even more painful, I'm only supporting half of the servers right now. They want me to do this for all of them before the end of the year. –  Tim Wilde May 3 '13 at 22:34
    
I realize this doesn't answer your question but, there's also a Group Policy Preference Client Side Extension setting for changing the local Administrator's password. For something failing as intermittently as you describe, you're going to need to "see" a failure from end-to-end. Not sure how to guide you to gather more details, but you need to. :) –  jscott May 4 '13 at 0:45
    
so the fix so far is to just rerun the scripts, right and eventually it works? Have you ever run the script against a single server and have it fail for non obvious reasons? –  tony roth May 7 '13 at 13:48
    
Also do you run this script from just one workstation or are you running it in a distributed fashion? Could you be overwhelming the network connection on the device running the scripts? –  tony roth May 7 '13 at 14:02

2 Answers 2

up vote 0 down vote accepted

Run WireShark on the client computer, log all the network traffic as the script runs.

That's going to be a lot of data, but given that you're talking about manually running it on 800 servers, it's not going to be much worse than that.

Look for failed DNS resolution, look at comparing the servers that worked and the servers that failed.

WMI supports logging, errors and debug information. It might be useful on the client or on some of the failing servers: http://blogs.technet.com/b/askperf/archive/2008/03/04/wmi-debug-logging.aspx

share|improve this answer
    
In this case I'm pretty comfortable with the idea that DNS is the problem, I'd just prefilter on the dns protocol to see it. Could be wrong but the way that he states the fact that it works sporadically kinda suggests it. –  tony roth May 7 '13 at 15:33
1  
Ok, update time. WireShark showed bad DNS; many of the servers in question had migrated from a legacy domain, and updated publicly and functionally like they were in the production domain. However, server settings/NIC had cached old DNS so it was not connecting with info available. Got that sorted, fixed by trying every possible domain, which only added about 5 hours to the scripts runtime unfortunately. –  Tim Wilde May 23 '13 at 0:13

Have you checked the Windows Firewall or 3rd party firewall software to confirm it's disabled or that it allows access to the server running the Powershell script? I'd also check the subnet mask and verify you have "File and Printer Sharing" enabled, on the failed clients.

share|improve this answer
    
Yes, that's the other problem. These arent the same 1600 every month, it just seems to be about a 5% error rate. There's about 92 consistent repeat offenders. I'll double check the file and printer sharing and get back with you. Something to note; other items do work. I can get all members of the local groups, for example, with no issues. –  Tim Wilde May 3 '13 at 22:28
    
Is the clock on each server in sync with one time source? –  Tim May 7 '13 at 1:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.