Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have 2 1TB disks not in any RAID configuration. I'd like files I need to store to be placed on one of the disks depending on the capacity of the disks, and when accessing the file I suppose I'd need to find the file via a database containing a file map, or by using a hash. Are there any Linux utilities that provide this, or should I just create a PHP script?

Thanks

share|improve this question
1  
It would be interesting if there was some software that let you "RAID" two directories on different filesystems thereby increasing local storage or streamlining redundancy between local and remote FSs. A php script would not be good for this. You really need a kernel-level driver or some added functionality within the filesystem to enable what you are talking about -- that or wrap everything you do in a php script. –  Puddingfox Apr 29 '11 at 18:54
    
Raiding file systems across remote systems already exists. On large scale you have GFS (and quite a few other proprietary ones). Coda is a smaller scale on which would properly work, but for this why would you bother? –  Ryaner Apr 29 '11 at 20:10
add comment

6 Answers

Considering how cheap a 1TB disk is, get another and create a RAID5.. Redundancy and storage.

share|improve this answer
4  
+1 Once you factor in programming and debugging time * how much your time is worth, it's almost certainly cheaper to just buy another drive and do RAID 5. –  Kendall Apr 29 '11 at 18:51
    
Can I do RAID 5 without affecting the data on the current disks? –  Tom Apr 29 '11 at 18:56
2  
No, you need to do a backup, create the RAID 5, than restore the backuped data –  Kedare Apr 29 '11 at 19:11
    
How can I create the backup when the only disks large enough to backup the data (700GB) are to be used in the RAID 5 array? Isn't that catch-22? –  Tom Apr 29 '11 at 20:33
    
@Tom, that's when it's good to have tech-savvy friends... who generally have gear they can barrow you. –  Chris S Apr 29 '11 at 20:42
show 1 more comment

Greyhole will distribute your files across multiple drives. It will also allow you to specify redundancy, so that certain files have redundant copies stored on multiple drives. It is targeted at the home server or workstation and not as a production enterprise solution.

share|improve this answer
    
That's interesting! code.google.com/p/greyhole/wiki/HowGreyholeWorks is a good read. –  sciurus May 1 '11 at 4:58
add comment

It sounds like all you care about is being able to utilize all 2TB of storage without having to manually place files on one drive or another. Either LVM or RAID0 can solve this problem for you at the expense of increased risk of failure. For LVM, you would make each 1TB drive an LVM physical volume and put them both in a single volume group. After that you could create logical volumes that up to 2TB in size. For RAID0, you'd just create the RAID device.

# pretending your unused 1TB disks are sdy and sdz
# for LVM
pvcreate /dev/sdy /dev/sdz
vgcreate myvg /dev/sdy /dev/sdz
lvcreate --name mylv --size 100%
# for RAID0
mdadm --create /dev/md0 --raid-devices 2 --level 0 /dev/sdy /dev/sdz

I don't know of a way to transparently merge separate filesystems into a single storage pool. This sort of sharding isn't uncommon, it's just typically implemented at the application rather than the storage layer. Engineyard has a paper describing filesystem sharding tactics and processes.

share|improve this answer
add comment

The right way is to use LVM.

Personally I just put most of my media collections on one disk, other things on the other disk.

share|improve this answer
add comment

If you plan to scale in future say to 10 hardrives across multiple servers consider using clustred filesystem such as http://en.wikipedia.org/wiki/GlusterFS

share|improve this answer
add comment

Three years late, but still relevant. I had the same problem and FakeRaid was simply out of the question. Use AUFS. It will join the drives under a single drive. The mfs setting will put new files on the drive with the most free space. There is also rr which is round robin and pmfs which will put files onto the drive that has the folder already and has the most free space. I personally use pmfs. My setup works like so.

The fstab:

# The Archive
LABEL=Archive_000 /mnt/Archive_000 ext4 defaults 0 0
LABEL=Archive_001 /mnt/Archive_001 ext4 defaults 0 0
LABEL=Archive_002 /mnt/Archive_002 ext4 defaults 0 0
LABEL=Archive_003 /mnt/Archive_003 ext4 defaults 0 0
LABEL=Archive_004 /mnt/Archive_004 ext4 defaults 0 0
LABEL=Archive_005 /mnt/Archive_005 ext4 defaults 0 0
LABEL=Archive_006 /mnt/Archive_006 ext4 defaults 0 0
LABEL=Archive_007 /mnt/Archive_007 ext4 defaults 0 0
LABEL=Parity      /mnt/Parity      ext4 defaults 0 0
LABEL=Q-Parity    /mnt/Q-Parity    ext4 defaults 0 0

I added the init.d script (Due to drive mount times being too slow to keep up with aufs mount):

d0="Archive_000"
d1="Archive_001"
d2="Archive_002"
d3="Archive_003"
d4="Archive_004"
d5="Archive_005"
d6="Archive_006"
d7="Archive_007"

mount -t aufs -o noxino -o br=$d0=rw:$d1=rw:$d2=rw:$d3=rw:$d4=rw:$d5=rw:$d6=rw:$d7=rw -o create=mfs -o sum none Archive

That gives me 10 mounts under /mnt. I like it this way because I use SnapRAID which you'll have to download and compile (there are guides for it). I use this on a Samba server, so the only thing everyone else sees is just the Archive folder. Make sure to make the directory other wise you'll get a mount error.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.