6

I have 2 1TB disks not in any RAID configuration. I'd like files I need to store to be placed on one of the disks depending on the capacity of the disks, and when accessing the file I suppose I'd need to find the file via a database containing a file map, or by using a hash. Are there any Linux utilities that provide this, or should I just create a PHP script?

Thanks

2
  • 1
    It would be interesting if there was some software that let you "RAID" two directories on different filesystems thereby increasing local storage or streamlining redundancy between local and remote FSs. A php script would not be good for this. You really need a kernel-level driver or some added functionality within the filesystem to enable what you are talking about -- that or wrap everything you do in a php script.
    – jamesbtate
    Apr 29, 2011 at 18:54
  • Raiding file systems across remote systems already exists. On large scale you have GFS (and quite a few other proprietary ones). Coda is a smaller scale on which would properly work, but for this why would you bother?
    – Ryaner
    Apr 29, 2011 at 20:10

6 Answers 6

9

Considering how cheap a 1TB disk is, get another and create a RAID5.. Redundancy and storage.

6
  • 4
    +1 Once you factor in programming and debugging time * how much your time is worth, it's almost certainly cheaper to just buy another drive and do RAID 5.
    – Kendall
    Apr 29, 2011 at 18:51
  • Can I do RAID 5 without affecting the data on the current disks?
    – Tom
    Apr 29, 2011 at 18:56
  • 2
    No, you need to do a backup, create the RAID 5, than restore the backuped data
    – Kedare
    Apr 29, 2011 at 19:11
  • How can I create the backup when the only disks large enough to backup the data (700GB) are to be used in the RAID 5 array? Isn't that catch-22?
    – Tom
    Apr 29, 2011 at 20:33
  • @Tom, that's when it's good to have tech-savvy friends... who generally have gear they can barrow you.
    – Chris S
    Apr 29, 2011 at 20:42
3

Three years late, but still relevant. I had the same problem and FakeRaid was simply out of the question. Use AUFS. It will join the drives under a single drive. The mfs setting will put new files on the drive with the most free space. There is also rr which is round robin and pmfs which will put files onto the drive that has the folder already and has the most free space. I personally use pmfs. My setup works like so.

The fstab:

# The Archive
LABEL=Archive_000 /mnt/Archive_000 ext4 defaults 0 0
LABEL=Archive_001 /mnt/Archive_001 ext4 defaults 0 0
LABEL=Archive_002 /mnt/Archive_002 ext4 defaults 0 0
LABEL=Archive_003 /mnt/Archive_003 ext4 defaults 0 0
LABEL=Archive_004 /mnt/Archive_004 ext4 defaults 0 0
LABEL=Archive_005 /mnt/Archive_005 ext4 defaults 0 0
LABEL=Archive_006 /mnt/Archive_006 ext4 defaults 0 0
LABEL=Archive_007 /mnt/Archive_007 ext4 defaults 0 0
LABEL=Parity      /mnt/Parity      ext4 defaults 0 0
LABEL=Q-Parity    /mnt/Q-Parity    ext4 defaults 0 0

I added the init.d script (Due to drive mount times being too slow to keep up with aufs mount):

d0="Archive_000"
d1="Archive_001"
d2="Archive_002"
d3="Archive_003"
d4="Archive_004"
d5="Archive_005"
d6="Archive_006"
d7="Archive_007"

mount -t aufs -o noxino -o br=$d0=rw:$d1=rw:$d2=rw:$d3=rw:$d4=rw:$d5=rw:$d6=rw:$d7=rw -o create=mfs -o sum none Archive

That gives me 10 mounts under /mnt. I like it this way because I use SnapRAID which you'll have to download and compile (there are guides for it). I use this on a Samba server, so the only thing everyone else sees is just the Archive folder. Make sure to make the directory other wise you'll get a mount error.

2

Greyhole will distribute your files across multiple drives. It will also allow you to specify redundancy, so that certain files have redundant copies stored on multiple drives. It is targeted at the home server or workstation and not as a production enterprise solution.

1
1

It sounds like all you care about is being able to utilize all 2TB of storage without having to manually place files on one drive or another. Either LVM or RAID0 can solve this problem for you at the expense of increased risk of failure. For LVM, you would make each 1TB drive an LVM physical volume and put them both in a single volume group. After that you could create logical volumes that up to 2TB in size. For RAID0, you'd just create the RAID device.

# pretending your unused 1TB disks are sdy and sdz
# for LVM
pvcreate /dev/sdy /dev/sdz
vgcreate myvg /dev/sdy /dev/sdz
lvcreate --name mylv --size 100%
# for RAID0
mdadm --create /dev/md0 --raid-devices 2 --level 0 /dev/sdy /dev/sdz

I don't know of a way to transparently merge separate filesystems into a single storage pool. This sort of sharding isn't uncommon, it's just typically implemented at the application rather than the storage layer. Engineyard has a paper describing filesystem sharding tactics and processes.

1

The right way is to use LVM.

Personally I just put most of my media collections on one disk, other things on the other disk.

0

If you plan to scale in future say to 10 hardrives across multiple servers consider using clustred filesystem such as http://en.wikipedia.org/wiki/GlusterFS

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .