RAM disk and physical RAID

Question

I need to create a lot of temporary files for which I need to have fast access, so I'm thinking about using a RAM disk. The problem is that the temporary files might get pretty big (1-4 GB) and in this case the disk-storage would be OK.

Is there any way to create a mix between a RAM disk and a physical HDD, so that the HDD gets used in the worst case (when I meet the big temporary files)?

---Edit---

I cannot afford faster SSD's, and I don't need persistent storage, and I don't need 100 GB of temporary data. I'm using my home computer for this (2 GHz AMD system...). I received a tip about UnionFS. What is the experience with it?

Answer 1

Use tmpfs and a big swap partition or file. This file system will cache data in memory as long as it can, and swap them to disk if they don't fit into RAM.

Answer 2

Why not try using the solid state SSD drives? You can get those with a capacity of 100 GB or more, and they are way faster than standard disks.

Answer 3

I'd second the option to do RAID 0 (stripe) with a hardware RAID controller. I quoted a video encoding server and the scratch drives are 15k SAS in RAID 0 and it chews through video like nothing else.

Answer 4

Depending on how long you expect to hold on to these temporary files for, you could use a filesystem such as XFS or ext4 which supports delayed allocation. Such filesystems are "clever enough" that if a short-lived file is deleted before it was allocated (still in the cache), then it never gets allocated on-disc at all and no data is written.

I tried this with XFS vs ext3 and the difference was amazing, a test program which creates lots of temp files and then removes them after a while, did almost no IO on XFS, but loads on ext3.

But I agree with other posters, tmpfs is pretty good too (actually may be better)

Answer 5

You're asking if you can create a ramdrive to cache some temp files and another partition of disk space for your big temp files and automatically mix and match between the two as necessary? This sounds a bit on the complicated side to achieve. You might be better off, depending on budget, at looked at installing an SSD drive to use for temp files/swap or look at getting a couple of hard disks and striping data read/writes.

I suppose you could try using software RAID with two drives, one physical partition and one being entirely in memory, to create a JBOD (just a bunch of disks); you could be asking for issues with that though since the one disk isn't real. It might yield some funny numbers in performance if it did work. Will it work? No idea. Haven't heard of people doing it but this would be the approach I'd research first.

Personally I'd look at upgrading your disks to faster drives (like 15,000 RPM) and using RAID to stripe them, software or hardware RAID (not on mobo) or using the SSD route, as those can be blazing fast but burn your wallet as well. Otherwise you could look at running a 64 bit OS and getting enough RAM to create a several-gig RAM disk to achieve what you're looking at doing.