2

We have two large storage servers (+100TB), one runs on ZFS the other one runs XFS, we intent to use the XFS as our work server and use the ZFS as the backup server (snapshots <3). Now the problem is keeping these beasts in sync ... (sync as in daily synced)

The easiest option is to use rsync, but sadly the directory structure is deep, and full of hard links, all over the place. So this means we would need to do a "global" scan which would take ages... On top of that, most of the data is created and never modified. So rsync might just not be the way to go.

I looked into inotify, which seems relatively cheap and since we only wanne sync on a daily base, would be able to offload to a good time... sadly, if we only look to the created files we would copy hard links as data which would double the amount of storage used in our backup ... (basically there is no way to do the -H check from rsync)

The only option left I could think off would be to reorganize our storage to use a date based directory, sadly moving around that much data is not something we would prefer ...

Are there other options ?

For reference :

  • Server with XFS has a raid controller (no JBOD option), and SATA disks (WD RE). 32Gb RAM
  • Server with ZFS has a HBA controller and SAS disks. 126Gb RAM

When I reference ZFS as beeing slow, I see 'ls' taking seconds...

  • 1
    Why not migrate the work server to ZFS as well. Then you can use the native ZFS send/receive mechanisms. – EEAA Aug 31 '16 at 16:30
  • Well we tried that, but ZFS with this amount of (meta-) data is just to slow on our setup... ZFS send/receive also looks like a full copy which is even more data over the line no ? – SvennD Aug 31 '16 at 19:29
  • No, with snapshots, send/receive is only changed blocks. – EEAA Aug 31 '16 at 19:40
  • Oh, I should test this feature better then, while I can't / won't go back to a double ZFS in this particular case, its handy to know! Thanks. – SvennD Aug 31 '16 at 19:45
  • Why are the server specifications so mismatched? – ewwhite Sep 1 '16 at 10:36
4

How fast is "fast enough"?

You are doing this once per day, so I'd suspect that if it takes 2-3 hours, that is sufficient.

In that case, "rsync -avP" should be all you need. The newest versions handle large directories, deep hierarchies, and doesn't require as much RAM as older versions.

If no files changed, "rsync -a" will be as fast as "ls -lR". You can't get any faster than "ls -lR" because it does an lstat() of every file on the system.

Benchmark "ls -lR" and "rsync -a". If they are slower than you think they should be, look at https://serverfault.com/a/746868/6472 for advice.

If you need something faster than the "ls -lR" benchmark, you'll have to either write something that uses "inotify", or use some kind of block-based system. In particular, using ZFS on both systems would enable you to use the snapshot export/import system that ZFS has built-in.

  • You double posted, but thanks for your input, I check inotify more closely and all-in-all it seems like a half ass solution to this problem. For the moment I think I'm going to try just rsyncing it until a better solution is setup (perhaps the popular ZFS/ZFS setup suggested here) – SvennD Sep 1 '16 at 20:01
  • @SvennD I check inotify more closely and all-in-all it seems like a half ass solution to this problem. I'd venture to say it's not even that good. It's event-driven, as TomOnTime noted. Which means if you base your backups off using inotify and miss an event, that data doesn't get backed up. So to ensure data integrity you have to do a state-based backup anyway, again as TomOnTime noted. So you really don't gain much. – Andrew Henle Sep 2 '16 at 12:48
  • We went with TomOnTime's rsync, as seemingly our tests where no representation of the real workload, when we sync only once a month the changes take up way over 10h but, syncing them daily/hourly seems to take way less time then anticipated, in the future, a ZFS<->ZFS might be a better solution, but at this point its not yet practical in our use case. Thanks all ! – SvennD Sep 3 '16 at 10:10
6

You really should be using ZFS on both sides, coupled with a block-level snapshot/replication routine like Sanoid.

Without that, you're stuck with file-based operations and the pain of the rsync file scan.

  • Excellent answer as always ewwhite, sadly ZFS is just to slow, perhaps its the workload or more likely the fact that we don't know anything about tuning ZFS properly... – SvennD Aug 31 '16 at 19:36
  • Then the question should be about ZFS tuning, which I can help with :) I've replaced all of my XFS production workloads with ZFS on Linux over the past 4 years. – ewwhite Aug 31 '16 at 20:14
  • @SvennD What hardware are you using? Especially disk controller(s) and disk types on your "slow" system(s)? Because rsync is going to be a lot slower than ZFS send/receive with complex file systems - rsync has to find what's changed while ZFS already has the difference set in the snapshot. – Andrew Henle Sep 1 '16 at 8:49
  • @AndrewHenle I added the hardware for reference, however, I doubt I can 'tune' ZFS enough to sell it again to he colleagues... – SvennD Sep 1 '16 at 8:56
  • 1
    I'm confused. It sounds like you are using the faster hardware for backups... something that runs once a day. Use the faster hardware as the regular server and the older hardware for backups. – TomOnTime Sep 1 '16 at 17:39
2

I would adopt a 2-part strategy... and at the end I'll suggest a 3rd part that is optional.

Part 1: Use inotify: Write a program that uses inotify to log which files were created, deleted, and modified. Write another program that reads the log, removes any duplicates, and does backups of those files (and deletes the deleted files). This won't be easy. Programming inotify is complex. The log can't be a simple text file since filenames can include newlines. If the system crashes while the log is being written, you'll need to be able to deal with partially-written filenames.

Part 2: Weekly rsync's just in case. Every few days do a "rsync -a --delete" to catch any files that were missed. The solution in part 1 is imperfect. If your program doesn't keep up with inotify, it may miss some files. If the machine reboots, the log of created/deleted/modified files may lose the most recent items. Bugs and other issues may also result in missing some files.

Optional Part 3: After you have this running for a few weeks and you've gotten all the bugs worked out, you'll still find that rsync is occasionally finding files that were missed. I promise you that will happen. inotify is "best effort". So, at this point you you'll realize that maintaining the code in Part 1 and Part 2 is twice as much work as you expected. To solve this problem, throw away the code you wrote in Part 1, because rsync is all you really needed in the first place.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.