What should I watch out for in building low-budget storage servers?

Question

I'm looking to replace our current storage. The existing system is just a 450GB file share on our 7yr old domain controller, plus another 150GB from another server. We map these to drives on the client machines. Both back up to a single 400GB tape drive. Add in a few system states and a sql server db and the nightly backup is up around 600GB. This is more than a single tape will do, and so we currently have a staggered backup where we only take a "full" from 1/5 of the total space each weeknight + a differential of everything.

To replace this, I want to go to two dedicated servers (primary, plus mirror and backup in one). The plan for the first is a RAID 1 volume for the OS + 2 RAID 10 2TB volumes for data (4 disks + 1 hot spare each, 12 1TB disks total including OS). The plan for the other is to have the same 12 1TB disks (mirrored from the first server each night). It will also have 2 6TB RAID 6 volumes (5 2TB disks each, separate volumes to improve the inevitable rebuild times) that I plan to use our existing backup exec software to make a differential snapshot each night. This will provide historical/generational storage for a true backup in place of our tape, and should be enough space to get about 30 days worth (a simplistic check of our change rate shows I need about a 3:1 ratio to accomplish this). The 2nd server will live on the far side of campus to avoid the need to take anything off site.

I'd much rather buy a vendor-supported solution, but cost is a huge issue here. I know I can do what I've described above for a total cost of around $13,000 (including OS licensing, thank you academic pricing for educational institutions), which places it way below any commercial option I've seen. I have a reliable partner for the basic chassis, RAID cards, and backplane, so I know I'm getting good equipment there, and I know I need a quality RAID card. I just haven't built a server of this scale before, and hence my questions:

Is there anything I'm missing that leaves me unnecessarily open to data loss? Anything I should look to do or avoid when assembling, installing, and configuring the servers? Nothing has been purchased yet, so I'm open to alternate suggestions — but the pricing is key. Some items that catch my eye right off the bat that I'd appreciate comment on:

• To reduce backup time/stress on the primary, I'd like to take the nightly backup from the mirror. But will copying to the mirror destroy the archive bit I need to get good differentials/incrementals?
• I'm not sure exactly how to configure the generational storage here, since both differential and incremental need a base to work from. I know I can use the primary storage directly for this, and that's not a big deal for a file here and there, but what if I need to do a complete rebuild of a volume? How do I make sure that files that don't change for 30 days are still available?
• The solution I just described requires 44TB of raw disks space, but only exposes 4TB to my users. The rest is lost to RAID, mirror, and historical backup space. Can I safely do better than that 11:1 ratio?

Answer 1

I'd be really wary of building your own computers. It sounds like you're pretty dead-set on it, though. Be sure to observe proper anti-static procedures when handing the equipment. Try to avoid using hard disk drives from the same manufacturing lot exclusively when assembling your RAID volumes. Be sure that the disks you're purchasing were shipped in the proper shock-resistant packaging as dictated by the manufacturer (and not a typically NewEgg "in their anti-static bags thrown in a box with some bubble wrap and peanuts" arrangement).

Since you're going to build your own box you need to stock sufficient spares for hard-to-replace components for the duration of the intended production life of the system. Having spare parts availability is part of what makes the "commercial" options. If you lose a backplane, RAID controller, or motherboard you need to be ready to replace them. You're acting as your own hardware support vendor. That's going to increase your purchase cost, but you should acquire spares now while they're still readily available.

I'd shake-down the hardware pretty severely before I even thought about using it in production. Any load-testing or production use simulation you can throw at the hardware is a good idea. You definitely want to shake out any "infant mortality" in the hardware before you go into production.

A "mirror" without multiple prior generations is a poor off-site copy. If you're doing the "mirror" to allow you to put the backup file server into production in the event of failure of the production then it makes sense to keep such a "mirror" up. It's not a backup, though, to my mind. That sounds more like a redundancy mechanism than a backup mechanism. I'd strongly recommend using a backup that allows for storage of multiple generations of changes on the remote server, even if you do also maintain a functional "mirror" of the data on the remote server for production failover purposes.

I'm glad to hear that you're still going to use tape. I'm a big fan of keeping backups off-site and offline. Offline backups are nice because their integrity (provided they're taken properly to begin with) is much easier to insure than a system that remains online all the time. It's pretty hard to attack the data on a tape in a locked box remotely.

Tape is also a great storage medium for archival storage. The cost to expand your retention windows is incremental (as opposed to adding more spinning media to a purely hard disk-based solution). I think you should strongly consider using tape to retain multiple generations of all of your data off-site and offline. Be sure that you're fulfilling any business requirements for maintaining long-term archives on tape and plan to purchase additional media as necessary. You can keep those archives in a more geographically diverse location to allow for some recovery capability in the event of a major physical disaster.

If you're doing a backup that's truly a "copy" then you're not going to lose archive bits. That depends on the software you're using.

Answer 2

Here's a suggestion regarding your backups that may help to reduce your storage needs:

Perform a Full SQL and System State backup every day (unless you're comfortable performing point in time SQL backups).

Switch to performing incremental backups on a daily basis for your file server backups. I've been in the IT field for 11 years and have yet to encounter a scenario (knock on wood) where I've needed a differential backup to restore a system or to restore data. I may need to go back through a few days of incrementals to find the "version" of a file that a user needs restored but that's rarely the case as the user usually wants the latest backup copy restored. In the event of a complete system failure, the worst case scenario is that you'll need to restore multiple incremental backup sets to bring the file server back in "sync". This will of course extend your recovery window, and that may or may not be acceptable to you and the organization.

I approach systems management from a "possibility vs. probability" point of view and in my experience, while it's possible that I may need a differential backup, it's not probable... from my experience. This has allowed me to reduce the size of our daily backups and the backup window as well.

Answer 3

There is appliance stuff with replication or at least remote differential copying features based on things like open-e, Nexenta or Windows Storage Server out there in the market.

On a whole, I'd rather not buy components to build them together into servers, but buy servers/appliances with system warranty - they would not cost significantly more, but might save you a lot of trouble in the case of a hardware defect.

Further, you should not use the tape "for SQL Server and system state backups" but to take copies of your critical data off site. You should see to it that you keep a history nonetheless - probably incorporating the SQL server into your standard backup process is a good idea.

If you are a Microsoft shop and eligible for academic licensing, you might want to look into Data Protection Manager to replace your aged BackupExec version - it would get you interesting features and good integration with Microsoft software.

To reduce backup time/stress on the primary, I'd like to take the nightly backup from the mirror. But will copying to the mirror destroy the archive bit I need to get good differentials?

Yes, it will. But differentials are a bad idea anyway - you probably don't need them. Take full backups at reasonable intervals and store them off site. Your backup strategy is primarily there to alleviate the effects of a catastrophic failure - if both of your campus IT sites are destroyed at the same time by a catastrophic event, it probably does not matter much if your users have lost one day or five days worth of data - as long as not the entire data is lost.

I'm looking at 44TB of raw disks space, but only exposing 4TB to my users after carving out RAID and backup space. Can I safely do better than that 11:1 ratio?

Depends on your security and performance requirements. You probably can save the cost for a RAID 10 setup on your "backup" mirror - as long as this is not meant to serve users directly, you can use a less performant and less expensive striping-with-parity setup (RAID 5, 6 or Z).

Answer 4

At Nexenta we have a number of excellent partners, such as Pogo Linux who have solutions that should fit your budget. Keep in mind they can help you come up with a scaled down solution, which will have necessary expansion capabilities to allow you to grow your SAN as needed. If you choose to build yourself there is the NexentaStor Community Edition option which I believe is currently capped at 12TB of disk capacity, which if you use deduplication and compression, where appropriate, could mean that you actually end up with far more data in that 12TB alloted capacity. However, you will not be able to upgrade to the commercial Enterprise edition should you choose to do that later. Also, if you have an uncertified solution support is always more of an issue than a partner solution. There are other options out there, but if you already have basic familiarity with Linux or better yet Solaris or Open Solaris, NexentaStor may prove to be an excellent option, since the underlying OS is robust and shares a lot with OpenSolaris.