I originally asked this on superuser but it occurs to me it's more more of a server topic.

I've just bought a 6-core Phenom with 16G of RAM. I use it primarily for compiling and video encoding (and occassional web/db). I'm finding all activities get disk-bound and I just can't keep all 6 cores fed. I'm buying an SSD raid to sit between the HDD and tmpfs.

I want to setup a "layered" filesystem where reads are cached on tmpfs but writes safely go through to the SSD. I want files (or blocks) that haven't been read lately on the SSD to then be written back to a HDD using a compressed FS or block layer.

So basically reads: - Check tmpfs - Check SSD - Check HD

And writes: - Straight to SSD (for safety), then tmpfs (for speed)

And periodically, or when space gets low: - Move least frequently accessed files down one layer.

I've seen a few projects of interest. CacheFS, cachefsd, bcache seem pretty close but I'm having trouble determining which are practical. bcache seems a little risky (early adoption), cachefs seems tied to specific network filesystems.

There are "union" projects unionfs and aufs that let you mount filesystems over each other (USB device over a DVD usually) but both are distributed as a patch and I get the impression this sort of "transparent" mounting was going to become a kernel feature rather than a FS.

I know the kernel has a built-in disk cache but it doesn't seem to work well with compiling. I see a 20x speed improvement when I move my source files to tmpfs. I think it's because the standard buffers are dedicated to a specific process and compiling creates and destroys thousands of processes during a build (just guessing there). It looks like I really want those files precached.

I've read tmpfs can use virtual memory. In that case is it practical to create a giant tmpfs with swap on the SSD?

I don't need to boot off the resulting layered filesystem. I can load grub, kernel and initrd from elsewhere if needed.

I'm leaning towards using ZFS with l2arc and zil on the SSD and zfs compression and dedup on the physical HDD drives.

So that's the background. The question has several components I guess:

* Recommended FS and/or block layer for the SSD and compressed HDD.
* Recommended mkfs parameters (block size, options etc...)
* Recommended cache/mount technology to bind the layers transparently
* Required mount parameters
* Required kernel options / patches, etc..
  • 1
    I only have a little experience with ZFS. In case you didn't now ZFS love lots of RAM (or L2ARC). I saw a video that said the default cache size is 80% of RAM (if that RAM isn't needed by something else), but that is tunable. Dedupe uses RAM so if you don't need hard drive space, you may want that RAM be used for caching. I linked to all seven videos on my blog scottmcclenning.blogspot.com/2010/10/zfs-cache-and-tuning.html If you want to learn about tuning ZFS, the videos and his blog should help. – Scott McClenning Oct 18 '10 at 1:12
  • I'm afraid you'll have to wait for stable btrfs to achieve this. – Hubert Kario Oct 18 '10 at 21:45
  • 1
    @Hubert: I'm not convinced of that. zfs-fuse appears to do it right now. the question is whether there's anything better – SpliFF Oct 19 '10 at 6:04
  • I'd say that the performance of zfs-fuse is unsatisfactory, especially when you want to use it for two tiered storage... – Hubert Kario Oct 19 '10 at 7:16
  • I hadn't used it, but it appears ZFS is being ported to Linux. zfsonlinux.org If ZFS isn't the answer, I believe it soon will be. – Scott McClenning Oct 20 '10 at 5:52
  • Compiling: What is the point in writing anything but the binaries to disk? Use tmpfs completely for this and then move the compiled stuff to your disk.

  • Video Encoding: I would suggest using rsync (in the mode that does transfer partial file differences) to tranfer files during encoding to disk. e.g. in a background loop that therminates when one video has been done. The Encoding itselv can take place in tmpfs, too.

BTW: What distribution do you use? SuSE and RedHat come with /dev/shm mounted (default: halv of your RAM with tmpfs).

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.