3

I'm designing a file server based around ZFS and I'm considering using USB flash drives in RAID 1 as boot devices. It seems that few people do this, and I'm wondering why. From what I see, using USB flash drives have some benefits:

  • Cheaper than hard drives and SSDs
  • Saves space and drive bays
  • May have higher MTBF than hard drives in this use case

Many servers have an internal USB port and can boot from the internal and external ports.

Comments about how MLC flash is limited to 10k-100k write cycles and may not be reliable enough are sure to come. I'm not too concerned about this, but perhaps I should be. File servers don't write to the OS disk very often, only for logs and such. Super Talent's flash drives are rated for 100k write cycles. Having two in RAID 1 and perhaps replacing one of them 3 years after set-up is still cheaper than alternatives and I believe would be reliable enough. And with ZFS, write errors will be detected quickly.

So why is this not more common?

3
  • How is "May have higher MTBF than hard drives in this use case" one of the benefits? Dec 7, 2009 at 0:29
  • John: They'll last longer? Dec 7, 2009 at 1:22
  • Thumb drives aren't cheaper than hard drives on a per GB basis. Dec 14, 2009 at 22:01

3 Answers 3

8

There are a bunch of reasons against this:

  • Flash tabs are unreliable; it's not just write cycles, the damn things just like to go on holidays regularly for any reason. When people said "we need a replacement for floppy disks", they obviously included "dies regularly" in the replacement criteria.
  • It's real easy to confuse which tab goes with which server; if you engineer your boot media right you can make this a non-issue (at the expense of providing a bunch of alternative problems), but any trivial implementation is going to result in tab-dependency, and then one day you boot a machine with the wrong tab and gaaaaaah...
  • Extra shenanigans are required in the BIOS of many machines to make this work right, and compatibility isn't assured. Buying 100 flash tabs only to find out they don't boot (even though your samples of the "same model" worked OK) isn't fun. "I see vendor giveaways in your future, young one".
  • It is stupidly easy to bump them out of place while the machine's running, since there's no "positive locking" to keep them in place. RAID-1 and very careful hardware techs help, but it's still a real risk. Reducing the number of thing hanging out the back of a server is high on my priority list of DC hardware "do's".

My previous job used flash tabs as boot devices while I was there; whilst it wasn't the worst part of their architecture, it certainly wasn't considered a plus. They've moved to an all-PXE boot architecture since I've left, and I think that's very much the right way to go.

1
  • +1 - If I'm going to say I agree with you, I'd might as well agree with you... >smile< Dec 7, 2009 at 1:13
3

Dell and HP are doing this, basically, with their embedded ESXi installations, albeit w/ SD memory cards in lieu of USB "flash drives". The readers are just attached to an internal USB connection inside the server's chassis.

It will remain to be seen if this is a viable strategy or not. These things haven't been on the market long enough to prove themselves (or to fail catastrophically).

The SD cards in dedicated readers have some distinct advantages, I think, to USB flash drives.

  • My personal experience with SD memory cards has been limited to about 10 cards with digital cameras, so I don't think it's really a representative sample of overall SD card reliability. They've been 100% reliable for me. I would think there would be very little writing in an ESXi embedded installation environment, as well. Having an OS that's tuned for booting off of flash would be a requirement, to me.

  • Since the slots are typically recessed (or sometimes inside the chassis) I think mixing up the media beteen server computers will be rare. Likewise, accidental removal will probably be rare, too.

    • The BIOS on server computers with built-in SD card slots should already be setup to boot from the SD card slot fairly easily, since the manufacturer put the device onto the machine for that purpose to begin with.

I agree with womble's points re: USB flash drives, except that I haven't had any reliability issues with them, personally. (Then again, I own a total of 3, none larger than 256MB. I'm just not that into them, I guess...) I definitely wouldn't boot an OS that wasn't designed for running off of flash in such an environment, either.

I also agree w/ womble that PXE booting is very convenient and generally The Right Thing(tm), but I'm not sure that SD-based installs of OS's like VMware ESXi in servers with purpose-built slots for this purpose is all that bad.

In summary, I'd be all for dedicated SD reader-based installs of OS's that don't have a heavy write footprint (like ESXi), but anti-USB flash drive-based installs or installs on any kind of flash media for OS's that aren't tuned for booting off of flash.

6
  • 1
    The OP is specifically asking about USB flash drives, though...
    – womble
    Dec 7, 2009 at 1:08
  • All three Dell PowerEdge servers I've looked at have internal USB ports with BIOS boot support. One drive could be inside and another at the rear. This is for a small business without the infrastructure to do PXE booting. Dec 7, 2009 at 1:19
  • For a small business, I'd say that avoiding non-standard stuff like USB flash drives is even more important. The supportability goes waaaay down, and leaving your successor with something unmaintainable is Bad Karma. Just spend the 5GB of disk space on a regular boot/root setup.
    – womble
    Dec 7, 2009 at 1:24
  • Software RAID5 (or RAIDZ in ZFS) can't be booted from, so one needs another 2 disks for the OS. Even though only a small amount of storage is needed, the cost for the smallest disks OEMs provide is still high for small jobs like this. Dec 7, 2009 at 1:50
  • 1
    VMware has deprecated use of SD cards/flash drives as boot devices due to a variety of issues, including write endurance, so it would seem this strategy has shown to be less than optimal.
    – Bob
    May 26, 2023 at 5:21
0

Well, we ran our HP Gen8 server from SD card (Vmware ESXi) and I can say that SD cards are very much reliable if you are installing the hypervisor in it. HP does recommend some specific SD cards to be used in their server.

If you look at the MTBF data of SD cads, we can find that the failure rates may be high if there are tons of write operation happens. But in the case of Vmware, the write operations are very minimal in the SD card.

There are many articles on internet which shows how to configure it. If you are stuck like me then you may probably look below provided link as well.

HP Gen8 Server: Unable to boot Vmware from SD Card http://www.tricksguide.com/hp-gen8-server-unable-to-boot-vmware-from-internal-sd-card.html

I also find lots of articles in Vmware web site. :)

SiRu

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .