5

So I have a backup server we've been using for some time, it's a FreeBSD server running zfs and serving it over NFS. The export is simple: /backup/vm -maproot=root -alldirs. If relevant, that was configured through zfs:

zfs get sharenfs
backup/vm sharenfs  -maproot=root -alldirs  local

It's been running fine and we've even restored these backups. Today I discovered purely by accident, that files read from the nfs share, don't match what was written (and what's on the server).

To demonstrate: on the server we have

pg11.txt (downloaded on the server)
pg11.txt.1 (uploaded by a client over nfs)

Both of which are Alice in Wonderland, downloaded from here: http://www.gutenberg.org/cache/epub/11/pg11.txt

On the nfs server:

md5 pg11.txt*
MD5 (pg11.txt) = eff1e5d84df1d3a543d1c578192a2367
MD5 (pg11.txt.1) = eff1e5d84df1d3a543d1c578192a2367

So far so good. Now on a client:

md5sum pg11.txt*
4d79d99b8eebe364cddf5ce42949bc3e  pg11.txt
eff1e5d84df1d3a543d1c578192a2367  pg11.txt.1

What? Reading pg11.txt from the client I can easily find lines like:

Alice started to her feet, for it flashed across her <80>^A^@<80>^V<A0>R+^@^@^@^A^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^A^@^@^@^A^@^@^A<A4>^@^@^@^A^^@^@^@^@^@^@^@^@^@^@^@^@^B<8E>^^@^@^@^@^@^B^B^@^@^@^@f7<D9>^@^@^@^@^@^@^V^V<EE>3^@^@^@^@^@^@^BFT^B<8C<FF>^E<D9>m(T^B<8C><E7>^]<CE>[<95>T^B<8C><E7>^]<CE>[<95>^@^A^@^@^@^@^@^@^@^A^@^@<U+FEFF>Project Gutenberg's Alice's Adventures in Wonderland, by Lewis Carroll

Now on a different client:

md5sum pg11.txt*
eff1e5d84df1d3a543d1c578192a2367  pg11.txt
b9c4076a85a151e730b9a9077fd6023b  pg11.txt.1

2nd client but over tcp:

md5sum  pg11.txt*
d80ce8c17092b1b759295e27a3c0af60  pg11.txt
14cde84fd05bd39845c9bb8fc0042eda  pg11.txt.1

The previous clients where both XenServer 6.2, if I try an Ubuntu system:

md5sum pg11.txt*
eff1e5d84df1d3a543d1c578192a2367  pg11.txt
81ca4f5b9b334d00a07fcb16f444a60a  pg11.txt.1

So every client seems to have a different picture, and usually not the right one. I'm hoping someone can give me some clue as to what's happening here and how to fix it, because I'm well stumped.

Edit:

The various files, including diff can be found here: https://gist.github.com/Whoops/0fbe1751675d5e344d43. It appears that the start of the file is repeated several (7) times, preceded by the same binary string each time. Also it's interesting to note that the corruption appears to be consistent for each client, i.e. each client always sees the same corrupted version, rather than different corruption on each read.

Edit2:

The problem occurs with both NFSv3 and 4. It appears to only occur on Linux clients, not other FreeBSDs. Tested clients are XenServer 6.2 and Ubuntu 10.04 which means if it's a client bug, it spans kernel versions 2.6 - 3.11. I don't currently have another FreeBSD server to test with.

  • Compare files with diff and you'll get the cue. – Kondybas Aug 31 '14 at 7:27
  • I believe that's a bug. – Michael Hampton Aug 31 '14 at 9:23
  • @Kondybas diff is why I was using Alice in Wonderland, since it's not great for VM backups :-p. The relevant files including diff can be found here: gist.github.com/Whoops/ec2bbffb73461283d5b4. Unfortunately, the corrupted version contains invalid UTF-8 so github, pastebin etc. refuse to display them. It appears the start of the file is repeated several times, followed by a binary string, but I still have no idea why. – Walton Hoops Aug 31 '14 at 16:07
  • You have to know what the difference between files, otherwise there is no possibility to find the answer. – Kondybas Aug 31 '14 at 16:09
  • @Kondybas is there something you are trying to lead me towards I'm not seeing? The difference (that I see) is that the corrupted version has the leading part of the file re-injected several times, always followed by the same binary string. Why this would happen is still beyond me. – Walton Hoops Aug 31 '14 at 16:14
3

Ok so it turns out this is a bug with the bxe driver in FreeBSD 10.0-Release specifically

The bxe(4) driver can cause packet corruption when TSO (TCP Segmentation Offload) feature is enabled. This feature is enabled by default and can be disabled by using a -tso parameter of ifconfig(8). It can be specified in rc.conf(5) like the following:

ifconfig_bxe0="DHCP -tso"

This bug has been fixed on FreeBSD 10.0-STABLE.

Many thanks to junovitch on the FreeBSD forums for figuring this out.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.