Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Is there any way I can verify the integrity of a remotely transferred file? I am currently transferring a large amount of files and folders from an ftp server to another remote server using wget, but I have no way of knowing if the files are corrupt. Is there any way I can verify the integrity of the transfer, by getting something like the MD5 hash of the remote files? Is there any other file transfer protocol that supports this?

share|improve this question

3 Answers 3

Depending on the tool you use it's possible to automatically hash and verify the downloaded files, the only tool that can do this that comes to mind immediately though is the DownThemAll! addon for Firefox.

What I normally use for situations like this a hash manifest file created by md5deep on Windows, you can use md5sum if you use Linux/UNIX or md5 on OS X. Using md5deep I CD to the folder containing the files to be transferred and run the command:

md5deep -l -r *>manifest.md5

On the remote end after transferring the files and manifest you would run the command:

md5deep -l -r -x manifest.md5 *

And it will show a list of every file that DOES NOT match the hash value in the manifest.

share|improve this answer
    
md5deep exists on Linux, too. I don't know what package it's in, but it's in the on my grml distribution version 0.9. –  Kevin M Dec 31 '09 at 23:10
    
I should have been more clear: right now, I dont have actual shell access to the source server. I only have ftp access. Maybe SCP. I might have to get the admin of the source server to send me an MD5 hash of the files –  Rafael Dec 31 '09 at 23:41
    
Persuading remote admins to do this has been a bit of a problem in the past year for me. Until I showed them an FDA requirement (we are a FDA certified development site) to do so they refused to spend time making hash manifests. Their argument was that they never had any problems downloading! Of course not, since they were on the same LAN as the server in question, little 'ol me logging in via the Internet does not have the save level of confidence in the ability of the network to deliver a perfect file every time. –  Ausmith1 Dec 31 '09 at 23:49

rsync is almost always the best answer for file transfers. It's best known for its differential optimization transfers (great when you already have a similar file or directory) but it profusely checksums every step of the transmissions.

share|improve this answer
    
Good answer, rsync does certainly take care of making sure that the files get to you in great shape. The only gotcha is proving to an auditor afterwards that the files you downloaded are the same files that the remote end has. Admittedly this is probably not an issue for Rafael who asked this question originally, but for me and those like me in regulated industries it is an issue and can only be solved with a hash manifest generated on both ends. –  Ausmith1 Jan 1 '10 at 19:02

While I really like the md5deep and rsync answers (and upvoted both), it sounds like you're facing a really hard group of people at the source server. An alternative that is really ugly but better than nothing is to cull the transfer log for the file sizes then compare them locally.

wget -nv -o log.txt ftp://ftp.myserver.com/welcome.msg

...created a logfile line that looks like this:

2010-01-01 09:47:17 URL: ftp://ftp.myserver.com/welcome.msg [470] -> "welcome.msg" [1]

So by using a little script-fu with some awk or whatnot, you can pull out the filename and filesize, then at least compare that they match. I reiterate that this isn't pretty and should be your last-ditch solution...but it works.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.