I have a powershell script I'm developing, which will download files from a HTTP server going through a proxy and including a UN/PW on the server.

I have files downloading fine, but some files are very large, and so I've been adding a function to check whether the file has changed (checking size). This works on the first file. But it times out on the second file at this line:

$test = $wc2.OpenRead($source) | Out-Null)

The code for my script, excluding credentials, is listed below. I'd appreciate advice as to how to properly close a connection (as I suspect is not happening) or on how to get a remote file's size via HTTP using Powershell.

Function getWebClient {
    $webClient = new-object System.Net.WebClient 

    $pxy = new-object System.Net.WebProxy $proxy  
    $pxy.Credentials = New-Object System.Net.NetworkCredential ("tnad\$proxy_un", "$proxy_pw") 
    $webClient.proxy=$pxy  
    $webClient.Headers.Add("user-agent", "Windows Powershell WebClient Header") 

    $WebClient.Credentials = New-Object System.Net.Networkcredential($un, $pw)

    return $WebClient
}

foreach ($ds in $datasetsTest) {
    Write-Host "Checking: $ds"
    $source = "$server$ds"
    $dest = "$destFolder$ds"

    #Test whether the destination file already exists, if it exists, compare file sizes.
    $destExists = Test-Path $dest

    if($destExists -eq $false) {
        Try {
            $wc1 = getWebClient
            $wc1.DownloadFile($source, $dest)
            $wc1.Dispose()
            Write-Host "File downloaded"  -foregroundcolor "green"
        } Catch {
            Write-Host "Downloading $ds failed..." -foregroundcolor "red"
        }
    } else {
        $localFileSize = (Get-Item $dest).length
        $wc2 = getWebClient
        $test = $wc2.OpenRead($source) | Out-Null
        $wc2.Dispose()

        $remoteFileSize = $wc2.ResponseHeaders["Content-Length"]

        Write-Host "Local file size: $localFileSize, remote file size: $remoteFileSize"
        if(!($localFileSize -eq $remoteFileSize)) {
            #file exists, but the server version changed...
            Try {
                Remove-Item $dest
                $wc3 = getWebClient
                $wc3.DownloadFile($source, $dest) 
                $wc3.Dispose($true)
                Write-Host "File downloaded"  -foregroundcolor "green"
            } Catch {
                Write-Host "Downloading $ds failed..." -foregroundcolor "red"
            }
        } else {
            Write-Host "Skipping, file exists and hasn't changed"  -foregroundcolor "magenta"
        }
    }
}
  • Not sure why this was closed-seems like something an admin would do – Jim B Sep 21 '15 at 18:13
  • It's totally what I, as an admin, would do. I need to get files off a share hosted by an external agency regularly, and it's most efficient to check if those files have changed before downloading them... – Alex Leith Sep 21 '15 at 22:22

I can't comment on this particular issue per se, but I'd like to offer some general recommendations. I've written some quite mission critical PS code over the last few years, some of which interacts with remote telemetry equipment over HTTPS. This code either has to succeed or fail gracefully, with very verbose logging. The code, although running in Windows VMs, is treated as an "embedded system / appliance".

I'd sincerely recommend adding error-handling code to your script. Granted, the Internet is full of bang-bang-bang examples of PowerShell code, with each successive step assuming that the previous one succeeded. This is sometimes made worse by over-use of PS pipelines, e.g.: enumerate-something | foreach-object{ do-something }, where some of the do-something's fail, meaning you're often left in an impossible roll-back situation.

Anyway, enough babbling. Recommendations:

  1. Enable strict mode. This'll prevent any unintentional type-casting / coercion. Pre-declare your variables, with correct type definitions.
  2. Use -erroraction silentlyContinue where possible, and trap errors with "if ($?)" checks... or wrap calls that generate exceptions with try{} catch{} constructs. Not all PS functions raise exceptions!
  3. Write plenty of log data
  4. Identify actions that are more likely to fail and code retry logic where possible.

I'll stop there!

Good luck.

  • Thanks, @Simon-Catlin, I think you're probably right. I have got something written in Python that achieves the need that this program is achieving, and over time, that script included all the error checking required. I guess what I'm saying is that you've got to start with something that works, and then get it working rigorously. There's no point having error checking if the function never works! – Alex Leith Sep 7 '15 at 22:14

You issue is due to the $test variable being assigned a null value.

When you build the variable $test

$test = $wc2.OpenRead($source) | Out-Null

the "| Out-Null" portion is nulling the results causing null to be assigned to $test, remove it and you should not have an issue.

As an aside for your question on closing System.Net.WebClient's connections: Action on you end is not required, all HTTP connections are closed when required and should not interfere with establishing a second connection. An open persistent connection will allow for a more timely and efficient connection on subsequent connections to the same URL. More information on persistent HTTP connections can be found here.

  • Thanks for your response, @Persistent13, but I don't think that changes anything. The pipe to Out-Null simply redirects what would usually be written to the terminal. The $test variable isn't ever used. Regardless, removing $test and also removing the pipe to Out-Null does not fix the timeouts. – Alex Leith Sep 7 '15 at 22:50

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.