- How do companies manage large file servers (e.g. 17 TB) and their related backups on a very tight-budget?
- Is it ok to use ZFS (or BTRFS) on a single virtual disk for its copy-on-write nature to eliminate the need for fsck? (i.e. not for its RAID, snapshot, etc. features)
We need to retire an ancient problematic storage system which was serving NFS storage for virtuals and is still our main file server. I now have a new 40TB iSCSI FreeNAS-based storage system and have already vMotion’ed all virtuals from old storage to new, but the 17 TB of SMB/CIFS & AFP shared files remain.
Backups are done via rsync which takes to long to scan 17 TB, so we’ve split into two volumes:
- 13 TB on a read-only “archive” volume for files that haven’t been modified for a year. To backup we simply make onsite and offsite copies using rsync - no need for daily backup versions.
- 4 TB live writable storage. This is backed up daily with rsync and uses hard-links to create “snapshots” / versions of the state of the file server on that particular day, so that we can recover overwritten files.
IT staff move files from archive to live when modifications are required, but these requests are now multiple times a day and no longer acceptable.
- Create a virtual Linux file server.
- Create 2x virtual disks on our new 40TB storage system. (13 TB archive + 4 TB writable live)
- Format above disks with ext4 file systems
- Use UnionFS with cow mount option to present a single unified view of above file systems to users, with all writes going to the 4TB writable system.
Problems with original plan:
- We are outgrowing the file-based nature of rsync as a backup tool, e.g. renaming a 1TB top-level folder results in a complete new copy of the entire 1TB folder. This would only add a few KB to a block-based backup.
- Implementing UnionFS just adds yet another layer of complication to the whole system, which I would really like to avoid.
- Due to rsync having to rescan the whole file server to compare changed files, it takes a very long time to complete backups, even if only a few files were changed.
- It requires us to do continuous archiving to keep writable volume small enough for backups to complete overnight.
Alternative plan 1: One large volume and ZFS snapshots for backups
- Create planned Linux file server above, but with one large volume, instead of two, eliminating the need for UnionFS
- To back up this very large server, which would take too long with rsync, use ZFS snapshots instead and use “zfs send” to replicate offsite.
Problem with alternative plan 1:
- fsck on a 17 TB ext4 file system would take days! Imagine a Monday-morning incident requiring a restart and forced fsck at boot, or worse, a file system corruption!
Alternative plan 2: ZFS/BTRFS file system on Linux server
- As per alternative plan 1, but use a copy-on-write file system such as ZFS or BTRFS instead of ext4, because these don’t require fsck.
Questions/concerns for alternative plan 2:
- Both ZFS and BTRFS want direct access to raw disks to implement their own RAID and are therefore not normally used on virtual disks. How well will this work on a single virtual disk?
Alternative plan 3: FreeNAS as file server directly
- Instead of having a separate virtual file server, share files from FreeNAS directly
Problem with alternative plan 3:
- We need to install various packages and custom perl/bash/python scripts to integrate this file server with our job tracking system. This will be fine on straight Linux, but I don't think it's a good idea on a pre-packaged FreeBSD-based ZFS storage system (FreeNAS). Updates may overwrite our changes, etc.
- Is alternative plan 2 - ZFS on single virtual disk - a good idea? If not, why?
- Can anyone suggest any better options?