I am looking for possible causes of file corruption.

I have some files that are 10s to 100s of MB in size that have gotten corrupted. Large chunks of the file have been overwritten with NULLs starting on some 64K boundry.

These files are not updated, they were written originally to local disk and then copied by a program using the CopyFile api to the SAN disk where they get read but never updated.

I expect that this is due to hardware errors but am not getting any error when reading the file (Just the nulls instead of the data that was there.)

  • 2
    I think you need to have a long chat with your storage admin. – Michael Hampton Oct 31 '14 at 13:46
  • We have but they are trying do deny that it could be them. – Arthur Oct 31 '14 at 14:18
  • 1
    Uh... OK. Is anyone actually buying that? "The files, which reside on your storage, are all effed up starting on some 64K boundary. Earn your paycheck and figure out why." – HopelessN00b Oct 31 '14 at 14:21
  • This is a case of outsourced hardware management. So they are claiming that the files must have been created with the NULLs. – Arthur Oct 31 '14 at 14:42

A few things I'd investigate would be:

  • Does your storage do zero page reclamation? If so, what size chunks do they work with? Does that line up with the damage? Not a smoking gun, but a definite clue.
  • Is the server a VM? What's the hypervisor? What are the settings for the virtual disk? Do you do any thin provisioning at that layer?
  • Does the backup software have access to this data read-only, or read-write? Have you looked in the logs for the software?
  • Do you know when the corruption occurred? Can you go back in your backups to a point before it happened? If so, this is the time you want to be scouring the logs of anything that might be a suspect.

It's really difficult to solve this forensically, but I'd see if you can find a way to reproduce the problem.

| improve this answer | |

I will offer this as a potential answer:

Has anyone started a large file restore from your back-up utility?

I saw something similar to this after a tech attempted to restore an entire network share.

Do you see any commonality among who owns the corrupted files?

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.