We have a cluster of web servers that currently stores temporary files (that have been uploaded by users) on a central shared filesystem. The limitation being that anytime the server containing the shared filesystem goes down our entire site is down (or crippled at least). I've looked at solutions like Gluster and GridFS to create a fault tolerant filesystem but none seemed ideal for ephemeral files that are frequently being created and deleted. Security is also important and Gluster didn't seem to have any means of encryption over the wire.
What are some good fault-tolerant distributed filesystem options that have the capability of being secure and handling frequent file creation / deletion?