16

Websites which supply ISO files for download will often give the md5 checksums of those files, which we can use to confirm that the file has downloaded correctly, and has not been corrupted.

Why is this necessary? Surely the error correcting properties of TCP are sufficient. If a packet isn’t received correctly, it will be retransmitted. Doesn’t the very nature of a TCP/IP connection guarantee data integrity?

  • 10
    Also don't forget about possible bugs in software and hardware doing the data transfer, on the endpoints as also inbetween. – sebix May 16 '15 at 12:16
  • The download may have terminated a few bytes early. You wouldn't necessarily notice it by the file size unless you are paying attention, and the TCP error correction would only have verified the part of the data that actually arrived. – Kevin Keane May 18 '15 at 0:03
  • Checksums may be handy but, in 20 years working with computers, I don't remember using it once. – Pedro Lobito May 18 '15 at 1:54
  • 2
    MD5 are hashes, not checksums. A checksum is used to check for errors, specifically bit errors during transmission. A cryptographic hash is meant to ensure that the data is exactly the same. In that sense a hash would be a superset of checksum, but they're not the same. Aside from that MD5 has been broken for 10 years now (see Wikipedia article, section Security). – 0xC0000022L May 18 '15 at 8:54
20

As has been noted by others, there are many possibilities for data corruption where any checksum at the transport layer cannot help, such as corruption happening already before the checksum is calculated at the sending side, a MITM intercepting and modifying the stream (data as well as checksums), corruption happening after validating the checksum at the receiving end, etc.

If we disregard all these other possibilities and focus on the specifics of the TCP checksum itself and what it actually does in terms of validating data integrity, it turns out that the properties of this checksum are not at all comprehensive in terms of detecting errors. The way this checksum algorithm was chosen rather reflects the requirement for speed in combination with the time period (late 1970's).

This is how the TCP checksum is calculated:

Checksum: 16 bits

The checksum field is the 16 bit one's complement of the one's complement sum of all 16 bit words in the header and text. If a segment contains an odd number of header and text octets to be checksummed, the last octet is padded on the right with zeros to form a 16 bit word for checksum purposes. The pad is not transmitted as part of the segment. While computing the checksum, the checksum field itself is replaced with zeros.

This means that any corruption that balances out when summing the data this way will go undetected. There are a number of categories of corruption to the data that this will allow but just as a trivial example: changing the order of the 16 bit words will always go undetected.


In practice, it catches many typical errors but does not at all guarantee integrity. It's also helped by how the L2 layer also does integrity checks (eg CRC32 of Ethernet frames), albeit only for the transmission on the local link, and many cases of corrupted data never even get passed to the TCP stack.

Validating the data using a strong hash, or preferably a cryptographic signature, is on a whole different level in terms of ensuring data integrity. The two can barely even be compared.

  • Best answer! I hate how the other answers mix up the concepts of cryptographic hashes and checksums. – 0xC0000022L May 18 '15 at 8:58
20

There is probably a zillion reasons why one should check the md5sum but a few do come to my mind:

  • Malicious activity - your ISO could have been tampered with on the way from the server
  • The page itself is spoofed (its best to have the md5sums signed as well :) )
  • Broken download (despite TCP error correction) (check this out)
  • ISO burnt incorrectly

And it only takes a few seconds anyway.

  • 21
    It also means that it's reasonably safe to download an ISO from a random mirror site, provided that you get the checksums from somewhere trusted; for example a PGP signed post to the foo-announce mailing list. – richardb May 16 '15 at 17:11
  • 2
    It's actually got nothing to do with protecting against malicious activity. If the ISO could have been replaced with a malicious one, so could the MD5 checksum value. Having them signed is a different matter but not what the OP is asking about. So instead of "malicious activity" being first on your list (it sure sounds good), it actually shouldn't even be on your list. You're giving people a false sense of security, which is dangerous. superuser.com/questions/849845/… – Austin ''Danger'' Powers May 18 '15 at 1:17
  • 1
    @Austin''Danger''Powers Umm, no, Konrad's right. For one, the download mirror usually is different from the site showing the checksum, and second, there's quite a lot of ISPs in the world that manipulate the traffic - TCP checksums will be fine, but you're downloading a different file. And of course, he's missing another point as well - the file might have been corrupted on the server, after the checksum was created. It happens all the time, especially for the more "hobbyist" servers (without proper RAID setups etc.). – Luaan May 18 '15 at 6:37
  • 2
    An answer from 2015 should advise against MD5 hashes. That algorithm has been broken for the last ten years (no exaggeration!). Also, you are mixing checksum and hash. They are two different things with different intent behind them. – 0xC0000022L May 18 '15 at 8:51
  • 1
    To add to add to the comment by @0xC0000022L SHA1 is best avoided if security is a major concern already too, though both it and MD5 are perfectly adequate to defend against accidental corruption. – David Spillett May 18 '15 at 10:20
6

TCP/IP does guarantee data integrity*. But it does not guarantee that 100% of a file has been downloaded. There can be many reasons why this could happen. For example: It is possible that you can mount an ISO that misses one or two bytes somewhere in the middle. You won't have a problem with it until you need one or two particular files that are corrupt. Comparing checksums ensure that you really did download the whole file.

* see comment

  • 8
    I think "does guarantee data integrity" is really over-selling what it actually does. It does make an attempt to check data integrity with a very lean approach, which is not particularly strong. – Håkan Lindqvist May 16 '15 at 13:31
6

The TCP checksum is only 16 bits. This means that, in the absence of other checksums, one out of every 65536 corrupted packets will be accepted as non-corrupted. If, for example, you were downloading an 8GB DVD image across a noisy link with a 1% corruption rate, you'd expect 81 undetectably-corrupted packets.

MD5 is a much larger checksum, at 128 bits. The odds of those 81 packets producing something with the same checksum as the original is about 1 in 1,000,000,000,000,000,000,000,000,000,000,000.

6

There are several reasons to verify the checksum of a file downloaded via HTTP:

  • Ensuring you received the entire file
    • Some clients, such as Firefox, may treat an interrupted connection as a successful download, leaving you with a truncated file but claiming it downloaded OK
  • Ensuring you received the correct file
    • e.g. a buggy, compromised or malicious server might send you something else
    • someone could tamper with the transfer (man-in-the-middle attack) - even HTTPS isn't safe from this if your system is compromised by e.g. Superfish, or the encryption method being used is weak
    • They might also just present you with a false download page, so you're not even connected to the real server (but in this case the checksums won't help much if you get them from the same fake server)
    • A number of ISPs have been caught injecting Javascript into pages in transmission for various reasons1; depending how well this is implemented, it might mangle some file downloads as well
    • A mirror might be hosting an out-of-date version of the file, or the admin might have uploaded the wrong file
  • Ensuring the file wasn't corrupted by something that TCP can't detect
    • e.g. the file could be corrupted on the server, so TCP will only ensure that the already corrupted file didn't get further mangled in transmission
    • or it could be corrupted after arriving on your end, by faulty memory/disk, buggy filesystem driver, etc
    • TCP checksums are only 16-bit, so the chances aren't astronomical (1 in 65536) that a corrupted packet won't be detected
  • With an ISO, ensuring that the disc burned correctly

1 sources in comment because lol rep

2

Daniel, Depending on the tool you are using for the ISO Download per say. If it is Say Firefox.. It may show the file download. However you may not have the full ISO intact. If you burn it then try to use it, information may be missing. This happens time to time on different webservers hosting files.

It is a good practice to at least compare the file size (total bytes or bits) make sure they match. Windows will show the file byte count different then say Linux. MD5 sum check will show same values no mater which OS is used. Hope this helps a bit. Cheers...

  • 2
    Windows shows the byte count differently from how Linux shows it? Really? I thought that abdomination went out with CP/M's file-size-as-blocks-count file system. (Now, if you're looking at something other than the byte count -- say, the file size display in Explorer -- it may well differ. But no sane sysadmin should be checking downloaded file integrity that way, so that's a non-issue.) Bytes are bytes. Looking at it in terms of bits makes no sense, though; when was the last time you downloaded and stored half a byte? – a CVn May 16 '15 at 19:52
2

I notice lots of interesting answers but there is a last thing to consider:Two Generals' Problem

The two generals problem and the Byzantine Generals problem consider specifically the implications of transferring information reliably through unreliable channels.

Checksums are just another layer of "increasing reliability", and one with a very slim chance of failure. This is the reason why it is so popular.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.