up vote 1 down vote favorite
share [g+] share [fb]

I have a 12TB enclosure that's filled with data (ext3). I'd like to make a copy of it, spanned across individual 3TB disks, each of which is readable on its own (no RAID or bonding). Ideally, I'd like this process to spit out some index of which files are on which disk, but that's not strictly necessary. The end result must be readable files on the disk in the original directory structure, not an archive file. Splitting directories (duplicating the directory structure) across volumes is fine.

The 3TB disks are not all mounted at the same time (and not able to be), so I need something that will let me pause the copying to switch disks. I strongly prefer a solution that doesn't require me to figure out which files go on which disk or keep track of what's been copied.

Is there an existing package that will do this?

link|improve this question
40% accept rate
Just to understand it better: you're looking to split the directory tree into 3 parts, each with a certain size. And the smaller structures have to be fully readable? You know it may never 'fully' fill up without heavily mixing small and big files from different parts in the structure? – Joris Sep 8 '11 at 17:52
With such requirements, please specify at least the operating system – Joris Sep 8 '11 at 17:53
Do you have at least 3TB of free space available somewhere for a big temporary file? – Zoredache Sep 8 '11 at 18:01
This is actually provably computationally hard if your 12T is really full and you've only got 4 3TB drives. Why? This is the bin packing problem rephrased. Welcome to NP-Hard :) en.wikipedia.org/wiki/Bin_packing_problem Now, if you've got ceil(11/9 * 4 + 1) (aka, 6) drives on hand, it's easy. – Bill Weiss Sep 9 '11 at 1:38
The individual files are sufficiently small (a few MB) that packing should not be a problem. If a few megs per drive are wasted, that's fine. :) – fields Sep 9 '11 at 16:37
show 1 more comment
feedback

2 Answers

up vote 1 down vote accepted

I don't know of an existing package that will do this but here is a completely untested no-error-checking script that may work:

#!/bin/sh

SRC=/foo
DST=/bar
IDX=/tmp/CONTENTS

DISK_ID=`uuidgen`

echo "$DISK_ID" > $DST/disk_id

cd $SRC
find . -type f | while read FILE; do

    FILE_SIZE=`stat -c '%a' "$FILE"`
    FREE_DST=`stat -c '%a' "$DST"`

    if [ $FREE_DST -lt $FILE_SIZE ]; then

            while [ $FREE_DST -lt $FILE_SIZE ]; do
                    read -p "Please mount a new disk at $DST and hit enter:"
                    FREE_DST=`stat -c '%a' "$DST"`
            done

            DISK_ID=`uuidgen`
            echo "$DISK_ID" > $DST/disk_id
    fi


    DST_DIR=`dirname "$FILE"`
    DST_DIR="$DST/$DST_DIR"
    [ -d "$DST_DIR" ] || mkdir -p "$DST_DIR"

    cp -p "$FILE" "$DST_DIR"

    echo "$DISK_ID $FILE" >> $IDX
done
link|improve this answer
love it! still a little risky, i would make sure that everything gets copied by looking at the combined used space on all 3 drives compared to the original – Silverfire Sep 9 '11 at 0:59
I was considering writing a ruby script to do it. This is pretty close. – fields Sep 9 '11 at 16:58
feedback
up vote 0 down vote

It's probably out of the scope you are looking for, but the free version of Unraid would do this. You could just assign the 3 drives as data with no parity drive. Set high water to "fill up" and set split levels to whatever meets your needs.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.