After very large file transfers (this only happens with 10gig+ files) to a specific system, the receiving system would have a mismatched checksum of the file.
The transfer seems to have happened sucessfully, the files match in terms of size to the byte, but the file will be corrupted. if it was a gzip it would have a crc error, and an md5sum would not match.
This is on the exact same server hardware running Ubuntu 10.04 64 bit.
There are no logs of memory issues, filesystem errors (ran an fsck) or hard drive failure.