2

Currently I create an NFS and mount it on a special server that does the actual backups and uploads it to s3. My main concern is I feel like overcomplicated this simple task. I schedule cron job to run the backup off hours but still would like to optimize and improve my strategy.

Currently the script on backup server will just find files on the NFS share within specific dates, tar them all up and use aws sync command to upload to s3 glacier. On the next run, the script gets a list of aws s3 objects and skips finding files for those dates (the date is part of archive file name, so its a very fast custom sync mechanism)

Checking network usage during the tar command over NFS is never more than 200Mbps and that is over lan through a 1Gbit switch. Upload to s3 is capped in aws cli configuration. So the script is very low key which is what I want. Yet something tells me maybe I should have just simply used aws sync to backup the data everyday and not try to reinvent the wheel. The extra complexity in find and tar and extra sync logic might lead to some unforseen bugs simply due to overcomplexity.

My question is what are the drawbacks of doing it my way (my main concern is is using NFS for heavy read backup operations bad practice?) And adding all that extra logic.

But if I just brute force it for simplicity and use aws sync for all files will take tremendously long for terabytes of data and also spike the CPU on the backup server and due to huge amount of read, spike CPU on production with the NFS share. I suppose I could try sync for specific dates and store a local file with what dates I already backed up.

Would appreciate some wisdom here :)

Improve this question
1
  • If you used something like zfs snapshots your diffs are tracked at the filesystem level so your tools don’t have to stat all the files. Not sure how to use zfs with aws as a target though.
    – Zoredache
    Dec 14, 2019 at 8:37

1 Answer 1

Reset to default
2

Sync programs like aws sync or rsync are easy to use, but metadata heavy. Large number of files causes a ridiculous number of metadata I/Os.

Your tar command customizes the sync to what you care about. Only checking mtime metadata, or traversing trees with dates in the name, will be more efficient. But yes, custom means you must ensure it is correct.

Block level backups don't bother with files, they backup the entire volume. However, incremental backups and restoring single files is trickier. Snapshot based might help (Linux LVM, ZFS). But you need to do this on the storage device, not a NFS share.

Whatever the implementation, always the most important part is restores.

Test a restore. Define a set of files, set a timer to the recovery time objective, and do it. Spot check the file list is complete, and with good integrity. Especially check edge cases, like the first and last files of the day.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .