Would it be at all feasible to use diskless nodes for cassandra? Im imagining a setup with one 'master node' that had all the appropriate software, disks, etc and the rest would PXE boot. The setup would do its compaction and write-back to a SAN (or possibly Lustre) based 10/40G iSCSI disk array.
When each node booted the primary would give it the appropriate config files based on MAC address (or similar) so it would know where on the SAN to get its initial data and where to write to.
No OS installation or maintenance. If a node runs into software or hardware trouble you just power cycle it. Assuming data redundancy > 1 you don't risk any loss this way either.
Tell me why this is a dumb idea.