Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Related to this question about using an SSD for system and HDD for data, except I would like my system to do this automatically...

Is it possible to have several layers of storage and push items automatically between them, using preferably free, open-source software?

I know of hugely expensive enterprise-class solutions like the EMC SAN->EMC Centera automatic archiving, but I was wondering if this sort of staged storage is possible to do automatically.

It would be nice to be able to have several layers in this : Memory->SSD->HDD->slower HDD or tape or some other archive solution.

Are there any filesystems which can do this automatically? (ZFS, Btrfs, HAMMER?)

Any Unix-variants are fine, as I'm interested in how this works and whether it's likely to be portable to Linux or other flavours (BSD etc).

share|improve this question

8 Answers 8

up vote 10 down vote accepted

Well, ZFS uses a storage layering called Hybrid Storage Pool (HSP):

  1. Layer: memory
  2. Layer: SSD-based read-caches (L2ARC) and write-caches (separate intent log aka slog).
  3. Layer: (cheap) harddisks

With HSP its easy to automatically benefit from the advantages of SSDs compared to a harddisk-only solution. A system using HSP can be both faster and cheaper than the latter. See this link for some nice examples and more details.

I think there are plans regarding hierarchical storage management (HSM) for ZFS (see for example the Automatic Data Migration (ADM) OpenSolaris project) but I don't know its current status.

share|improve this answer
    
This is just what I was thinking of, thanks! (Now to help on the Linux port... ;) –  David Gardner Jul 23 '09 at 8:30

TIER seems to answer your needs It is a linux kernel module that can create a tiered storage. It seems to learn by itself the pattern and optimize the placement into the storage

http://www.lessfs.com/wordpress/?p=776

share|improve this answer
    
Just saw the question is old ... Sorry ^^' –  Dolanor Aug 8 '12 at 15:33
2  
Revitalising old questions is better than having then rotting with old and out-of-date information on them! –  David Gardner Aug 9 '12 at 10:57
    
I'm worried that this guy wants to reduce redundancy for the sake of extra capacity (especially when SSDs are comparatively small and unreliable). –  Tobu Nov 6 '12 at 10:35
    
Also worried that he suggests raid as the solution, when experimental software needs backups most. –  Tobu Nov 6 '12 at 10:54
    
Here's a bit of more positive feedback though: permalink.gmane.org/gmane.comp.file-systems.ceph.devel/8316 –  Tobu Nov 6 '12 at 10:55

Check out the LVM based "lvmts" (LVM Tiered Storage) solution this guy is cooking up:

https://bbs.archlinux.org/viewtopic.php?pid=1140640#p1140640

Pretty cool.

share|improve this answer
    
+1 This looks very interesting -- I shall be keeping an eye on it! –  David Gardner Aug 6 '12 at 11:26

Answering my own question with something I just found:

I was just updating the kernel and looking at the new stuff that has been added, and there is now a 'CACHEFILES' option which allows for caching (usually remote) filesystems to a local filesystem. I guess I could use this to cache a slower storage mechanism (HDD) to a faster one (SSD), at least for one level of hierarchy.

share|improve this answer
    
Did you try this solution? How does it perform? –  skolima Nov 24 '09 at 16:17
1  
Nevermind - I found out myself that CacheFS for now only supports NFS and AFS. –  skolima Nov 24 '09 at 16:22
    
Using network filesystems for this isn't a bad idea, they tend to mature much faster than kernel patchsets. I wonder if Ceph or DRBD have useful modes for tiered storage. –  Tobu Nov 6 '12 at 10:48

Some relevant links. YMMV.

http://code.google.com/p/fscops/ -- "Online Hierarchical Storage Manager (OHSM) is the first attempt towards an enterprise level open source data storage manager which automatically moves data between high-cost and low-cost storage media.".

http://www.tack.ch/unix/dmapi/ -- XFS + DMAPI under Linux

http://jfs.sourceforge.net/ -- JFS + DMAPI under Linux

share|improve this answer
    
The OHSM project looks like an interesting one to follow. Only ext2 and the 2.6.30 kernel is supported for now, but I expect if it works out well then this will gain more filesystems. –  David Gardner Jan 22 '10 at 11:41
    
That project looks abandoned, most recent activity was in 2009. –  sendmoreinfo Sep 12 '11 at 22:05

RAID controllers offer some of these features.

"Leverages SSDs in front of HDD volumes to create high-capacity, high-performance controller cache pools"

http://www.lsi.com/channel/products/storagesw/Pages/MegaRAIDCacheCadeSoftware2-0.aspx

share|improve this answer

LVM2 snapshots come to mind... but you can't really do more than a single snapshot.

share|improve this answer

SAM-QFS is Sun's existing product and was open-sourced last year. It's CDDL, so you could only directly port it to *BSD.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.