2

I have the responsibility for changing the local administrator account password on my environment of 16,000 servers; I wrote a PowerShell script, but it took too long, so I added multi-threading using the powershell runspace factory to break out the 16,000 into manageable pieces.

There's a ~5% error rate (~800 servers); of these, 75-100 are clear errors that can be troubleshot (username not found, access denied, etc.), and 700-725 get the error message "The network path was not found".

However, pinging the servers gets a response and server engineers tell me they are operational, I have access, and that both PowerShell and WMI are running and functioning.

I have no idea where to begin troubleshooting. Here's the logic and code I'm using:

I use FQDNs, however my company tends to have servers listed differently in DNS to their FQDN, and the 2 choices will not resolve to one another. so servera.production.active.directory will not resolve to servera.mycompany.com. This function determines a valid FQDN for connecting and setting the password, returning either a valid FQDN or an empty string:

function get-validfqdn([string]$server, [string]$domain){

    $fqdn = $server + "." + (get_FQDN $domain)
    $altdn = $server + ".mycompany.com"

    if(Test-Connection -count 1 -computer $fqdn -quiet -TimeToLive 80){
        $valid = $fqdn
    }
    elseif{
        $valid = $altdn
    }else{
        $valid = ""

    return $valid
}

I attempt to execute the password change using the following code, embedded in a module and performed for each server in the list we're processing (this is a long function due to the PowerShell runspace factory code).

function Set-ServerPass([string]$filepath){

    $servers = Import-CSV $filepath
    $results = @()

    foreach($server in $servers){

        $svr = $server.Server
        $password = $server.Password
        $domain = $server.domain
        $fqdn = get-validfqdn $svr $domain

        if ($fqdn -ne ""){
            Try{
                $admin = [adsi]("WinNT://$fqdn/Administrator, user")
                $admin.psbase.invoke("SetPassword", "$password")

                $result.Error_Code = "0"
                $result.Error_Msg = "The operation was sucessful"

            }Catch{
                $error_msg = Trim_ExceptionMessage $_.exception.Message

                $result.Error_Code = "1"
                $result.Error_Msg = $error_msg
                $results += $result
            }
        }else{

            $result.Error_Code = "51"
            $result.Error_Msg = "The remote computer is not available"
        }
    } 
    return $results
}

Notes: Test-Connection filters out servers that would otherwise be unavailable; a timeout on this function defaults to ~180 seconds (3 mins x 1600 servers = too long).

This code works on 95% of the servers, and reports accurately after a year of running this script. However, server engineers are starting question whether this script works because when I report problems they're not seeing how or why I would get "the network path was not found" error when all of their tests say it is working fine.

Troubleshooting steps so far:

  • Run on different computer
  • Run as different admin
  • Run at different times of day - this was to prevent possible server activity from interrupting the script (application patching, reboots, etc)

This last 2 months I have manually troubleshot each of the 800 servers and ran the script some 15 times on just the failed servers. Re-running nets me about 10-300 more passwords being reset, but doesn't catch all of them, and it is very inconsistent.

On 3 occasions the server engineers reported no problems, I re-ran the script and it reset all of them with no errors.

So my questions are: what could be causing the error, and what should I look at to determine the the root cause? Settings on server? Settings on my workstation?

Setup is as follows: Windows XP Pro SP3. Servers are Windows Server 2003 or Windows Server 2008 R2. These errors occur on both server operating systems.

10
  • yikes that sounds painful, we have more devices then this and we just disable the administrator account! Truly not for everyone but once you have the process down for recovery its not that bad. Other then that this is quite an interesting problem. In this case I suspect DNS for some reason.
    – tony roth
    May 3, 2013 at 20:42
  • Even more painful, I'm only supporting half of the servers right now. They want me to do this for all of them before the end of the year.
    – Tim Wilde
    May 3, 2013 at 22:34
  • I realize this doesn't answer your question but, there's also a Group Policy Preference Client Side Extension setting for changing the local Administrator's password. For something failing as intermittently as you describe, you're going to need to "see" a failure from end-to-end. Not sure how to guide you to gather more details, but you need to. :)
    – jscott
    May 4, 2013 at 0:45
  • so the fix so far is to just rerun the scripts, right and eventually it works? Have you ever run the script against a single server and have it fail for non obvious reasons?
    – tony roth
    May 7, 2013 at 13:48
  • Also do you run this script from just one workstation or are you running it in a distributed fashion? Could you be overwhelming the network connection on the device running the scripts?
    – tony roth
    May 7, 2013 at 14:02

2 Answers 2

0

Run WireShark on the client computer, log all the network traffic as the script runs.

That's going to be a lot of data, but given that you're talking about manually running it on 800 servers, it's not going to be much worse than that.

Look for failed DNS resolution, look at comparing the servers that worked and the servers that failed.

WMI supports logging, errors and debug information. It might be useful on the client or on some of the failing servers: http://blogs.technet.com/b/askperf/archive/2008/03/04/wmi-debug-logging.aspx

2
  • In this case I'm pretty comfortable with the idea that DNS is the problem, I'd just prefilter on the dns protocol to see it. Could be wrong but the way that he states the fact that it works sporadically kinda suggests it.
    – tony roth
    May 7, 2013 at 15:33
  • 1
    Ok, update time. WireShark showed bad DNS; many of the servers in question had migrated from a legacy domain, and updated publicly and functionally like they were in the production domain. However, server settings/NIC had cached old DNS so it was not connecting with info available. Got that sorted, fixed by trying every possible domain, which only added about 5 hours to the scripts runtime unfortunately.
    – Tim Wilde
    May 23, 2013 at 0:13
0

Have you checked the Windows Firewall or 3rd party firewall software to confirm it's disabled or that it allows access to the server running the Powershell script? I'd also check the subnet mask and verify you have "File and Printer Sharing" enabled, on the failed clients.

2
  • Yes, that's the other problem. These arent the same 1600 every month, it just seems to be about a 5% error rate. There's about 92 consistent repeat offenders. I'll double check the file and printer sharing and get back with you. Something to note; other items do work. I can get all members of the local groups, for example, with no issues.
    – Tim Wilde
    May 3, 2013 at 22:28
  • Is the clock on each server in sync with one time source?
    – Tim
    May 7, 2013 at 1:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .