Dell and HP are doing this, basically, with their embedded ESXi installations, albeit w/ SD memory cards in lieu of USB "flash drives". The readers are just attached to an internal USB connection inside the server's chassis.
It will remain to be seen if this is a viable strategy or not. These things haven't been on the market long enough to prove themselves (or to fail catastrophically).
The SD cards in dedicated readers have some distinct advantages, I think, to USB flash drives.
My personal experience with SD memory cards has been limited to about 10 cards with digital cameras, so I don't think it's really a representative sample of overall SD card reliability. They've been 100% reliable for me. I would think there would be very little writing in an ESXi embedded installation environment, as well. Having an OS that's tuned for booting off of flash would be a requirement, to me.
Since the slots are typically recessed (or sometimes inside the chassis) I think mixing up the media beteen server computers will be rare. Likewise, accidental removal will probably be rare, too.
- The BIOS on server computers with built-in SD card slots should already be setup to boot from the SD card slot fairly easily, since the manufacturer put the device onto the machine for that purpose to begin with.
I agree with womble's points re: USB flash drives, except that I haven't had any reliability issues with them, personally. (Then again, I own a total of 3, none larger than 256MB. I'm just not that into them, I guess...) I definitely wouldn't boot an OS that wasn't designed for running off of flash in such an environment, either.
I also agree w/ womble that PXE booting is very convenient and generally The Right Thing(tm), but I'm not sure that SD-based installs of OS's like VMware ESXi in servers with purpose-built slots for this purpose is all that bad.
In summary, I'd be all for dedicated SD reader-based installs of OS's that don't have a heavy write footprint (like ESXi), but anti-USB flash drive-based installs or installs on any kind of flash media for OS's that aren't tuned for booting off of flash.