I'm scripting a full backup solution in powershell on Windows server 2016 for Hyper-V VM's and have run into the issue that calculating the MD5 hash over the network on a mounted network drive across a WAN is about half the speed of just doing the file copy itself over the WAN link. I've tried certutil.exe from the OS referencing the file across the share and it is extremely slow. I have also tried Get-FileHash in powershell with similar results. I have hundreds of 250MB files and have not been able to find a more efficient way of doing this without invoking a remote shell to run the command. Is there a better approach for this?

  • 1
    You're scripting a full backup? It sounds like you're reinventing the wheel. Have yo considered using one of the countless backup products/solutions that are available? – joeqwerty Jul 15 '19 at 20:54
  • It's at a customers request that everything would be within powershell and native OS utilities. It works, just REALLY slow comparing MD5 backups over a 20Mbps link with tons of large files. They do not want to be reliant on 3rd party vendors for anything. – Brad Jul 15 '19 at 20:59
  • The Windows Server Backup feature is completely free and supports Hyper-V. Is that what you're scripting in Powershell? – joeqwerty Jul 15 '19 at 21:14
  • Most likely the hash calculation reads such small blocks from the file that the protocol overhead becomes quite big, which degrades performance. Also the reading pattern of reading a block, calculating hash, reading next block makes it slower due to latency. In Linux I would use cat to print the file to md5sum. This will separate the file reading from calculating the hash value. – Tero Kilkanen Jul 15 '19 at 21:31
  • Tero, This is on Windows not NIX or I would just use rsync which would do all the lifting and verifying. – Brad Jul 16 '19 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.