Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Trying to import 53,800+ individual files (messages) using Gmail's POP fetcher.

Gmail understandably refuses, giving the error: "Too many messages to download. There are too many messages on the other server."

The folder in question looks like similar to:

/usr/home/customer/Maildir/cur/1203672790.V57I586f04M867101.mail.net:2,S
/usr/home/customer/Maildir/cur/1203676329.V57I586f22M520117.mail.net:2,S
/usr/home/customer/Maildir/cur/1203677194.V57I586f26M688004.mail.net:2,S
/usr/home/customer/Maildir/cur/1203679158.V57I586f2bM182864.mail.net:2,S
/usr/home/customer/Maildir/cur/1203680493.V57I586f33M740378.mail.net:2,S
/usr/home/customer/Maildir/cur/1203685837.V57I586f0bM835200.mail.net:2,S
/usr/home/customer/Maildir/cur/1203687920.V57I586f65M995884.mail.net:2,S
...

Using the shell (tcsh, sh, etc. on FreeBSD), what one-line command can I type to split this directory full of files into separate folders so Gmail only sees 1000 messages at a time? Something with find or ls | xargs mv maybe. Whatever is fastest.

The desired output directory would now look something like:

/usr/home/customer/Maildir/cur/1203672790.V57I586f04M867101.mail.net:2,S
/usr/home/customer/Maildir/cur/1203676329.V57I586f22M520117.mail.net:2,S
...
/usr/home/customer/set1/ (contains messages 1-1000)
/usr/home/customer/set2/ (contains messages 1001-2000)
/usr/home/customer/set3/ (etc.)
share|improve this question

2 Answers 2

up vote 4 down vote accepted

It's not a one-liner, but you could copy chunks like this (in csh):

foreach file (`ls | head -n 1000`)
mv $file /tmp/new/dir
end

I'm not 100% sure that pipe will work with the number of files you've got, but it's worth a shot. Also, you might be able to do 500 at a time with this command, just change that 1000 to 500.

share|improve this answer
    
Thanks very much Chris - this worked great. –  ane Mar 11 '10 at 20:14
2  
in bash: for file in (ls | head -n 1000`) do mv $file /tmp/new/dir; done` –  warren Mar 11 '10 at 23:09
    
Thanks Warren. That will work too. –  ane Mar 12 '10 at 15:27
count=target=0;

find srcdir/ -type f |
    while read file; do

        count=$((count+1));
        target=$((count/10000));

        [ -d $target ] || mkdir $target

        echo mv "$file" $target; #remove the 'echo' if you like what you see
    done

collapsed to a single line (and with the 'echo' safeguard removed):

count=target=0; find srcdir/ -type f | while read file; do count=$((count+1)); target=$((count/10000)); [ -d $target ] || mkdir $target; mv "$file" $target; done

This isn't the fastest but it's clean. this solution avoids parsing the output of 'ls' http://mywiki.wooledge.org/ParsingLs and quotes the references to "$file" otherwise files with abnormal names (as Maildir files often are) would break the code. For example if any of the files have spaces or semicolons then referencing $file without quotes isn't going to get you far (most of the time)

Read more about quoting: http://mywiki.wooledge.org/Quotes and http://wiki.bash-hackers.org/syntax/words

share|improve this answer
    
Thanks Stuart ! –  ane Jun 16 '10 at 12:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.