Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a folder full of 600gb of files. I want to automatically copy the first 300 to one folder, and the rest to another folder. I am not sure how to limit the results with ls or whichever so I can pass it as an argument...

platform is linux...

edit: I want to move 300gb, not the first 300 files. File sizes are arbitrary, and ordering does not matter.

share|improve this question
    
"first 300"? 300 files, or 300GB? –  Chad Huneycutt Jul 30 '09 at 15:49
    
It seems, first 300gb, if i get the english right. –  nik Jul 30 '09 at 15:51
1  
Are you wanting to move 300 files (as in the answer below) or 1/2 of the size of files? If so: - How many files are there? - Is the size distribution even across the files? –  CK. Jul 30 '09 at 15:51
1  
Oh, and what do you mean by 'first'? Is there some ordering you want, or do you want to just split the directory into two smaller ones? –  CK. Jul 30 '09 at 15:53
    
first 300gb, sorry. file sizes are aritrary –  Bill Gray Jul 30 '09 at 15:54

9 Answers 9

up vote 5 down vote accepted

Update: Oh, the first 300GB, well then... this probably slow, depending on file size, but I like the exercise :-)

filesize=0
for i in *; do 
    filesize=$(stat -c "%s" "$i");  
    (( totalsize += filesize )); 
    if [[ $totalsize < 322122547200 ]]; then
        mv "$i" first_300/
    else
        mv "$i" the_rest/
    fi
done

Hopefully there are not problems with the size of the int.


If you mean break them up into folders each with 300 files, maybe you want something like the following:

folder=0
counter=0
for i in *; do 
    mv $i foo_$folder/
    if [[ $(( counter % 10 )) -eq 0 ]]; then 
       (( folder++ ));
    fi
    (( counter++ ))
done

Although that might not be as fast as some of the find commands. If you just want to do the first 300 command, you could use the same counter strategy but use a while $counter -le 300.

share|improve this answer
1  
For fairly current bash shells integer math should be able to handle (2^64)-1 without wrapping around. If sizing is a problem then you can use du -k $i | awk '{print $1} in place of the stat statement. If you still want to use stat you may want to try counting the blocks used instead of bytes. This would mean using %b in place of $s as the format argument. 'stat -c "%B" /mountpoint', will tell you your blocksize for scaling. –  Rik Schneider Jul 30 '09 at 16:48

This is a way to get nearly equal 300gb distribution,

You could do a du based search to find distribution across top level directories and files and then split them into nearly two parts with some trials.

find . -maxdepth 1 -type d -exec du -sk {} \; | sort -n -k 1 > list.txt

This will give a sorted list of KB sizes.
You could do a trick like picking up alternate lines on this list for a quick nearly-even distribution

awk '{if (FNR%2==1) print $2}' list.txt > list1.txt
awk '{if (FNR%2==0) print $2}' list.txt > list2.txt

A Very rough distribution...

Finally, if you have very uneven file or directory sizes -- quite far from 300GB distribution,
keep yourself away from the bin-packing problem and do some simple trials on moving around a couple of lines between the two list files.
Find the difference between the two sets (with du) and move a directory/file
that is about half the difference from the larger list to the smaller one.
That should get you quite close

share|improve this answer
3  
Anything other than rough would mean solving discrete knapsack, so let's hope... :) –  Anders Eurenius Jul 30 '09 at 16:37
    
@Anders Eurenus - +1 exactly what I was thinking. "Congratulations, you've just invented the Knapsack problem." Fortunately, 300GB is probably just an approximation or an upper limit, so a greedy heuristic is probably fine. –  ConcernedOfTunbridgeWells Jul 30 '09 at 16:43
    
This is actually a bin packing problem with two bins. I think, Kyles answer can be tweaked to use a list sorted for size (largest first) to give quite an optimal solution. This answer tries something similar with the sorting. It is probably simpler and slightly less efficient. –  nik Jul 31 '09 at 2:31
    
I think, it is less efficient than a tweaked version of Kyles answer. But, probably better than his present unsorted version. –  nik Jul 31 '09 at 2:32

You could do it with find, head & xargs. It should look like this:

find ./ -type f -print0 | head -300 | xargs -0 -I mv {} /one/folder
find ./ -type f -print0 | xargs -0 -I mv {} /another/folder
share|improve this answer
2  
Was this tried on a shell in any way? –  nik Jul 30 '09 at 15:50
1  
You may have to do it this way: ./ -type f -print0 | head -300 | xargs -0 -I myvar cp myvar ./destfolder/ –  Maxwell Jul 30 '09 at 16:17
    
@Nik: Not at all. I only have AIX machine right now. This is why I said "should". –  Benoit Jul 30 '09 at 17:10
    
What's your reasoning for using "find", aside from the fact that it will search the directory tree? The OP says that the files are all in one folder, so I would expect that "ls" would be a faster way. Since the OP was trying to divide up the directory by size, "ls -l |sort -n -k 5" would be a good starting place. –  Ernie Jul 30 '09 at 18:16
    
@Ernie: It has been said that the folder contains all the files to move not that it contains /only/ these files... –  Benoit Jul 31 '09 at 7:48

WARNING! When you start calculating file sizes, you are likely to make the mistake to measure them by bytes, while most file systems will allocate disk space in blocks. And this block size varies from disk to disk but is often a multiple of 512.

Basically, that means you can have 500 files of one byte each, which would only be 500 bytes. But a file-system that allocates 2048 bytes per block would thus claim about 1 megabyte of disk space. Yeah, that's a lot of overhead.

Basically, you should round up the file-sizes you get by the block size of the file system you use. That way, you can measure them more precisely.

Then again, how much difference could it be? If the block size is 2048 bytes then the average amount of bytes "lost" would be 1 KB. With 300 files this would be about 300 KB that you would need more on top of your total size. You want to copy 300 GB but how many files would that be? And are the two disks using the same file system with the same block size?

Anyway, the error margin depends on the average file-size. If you have a lot of huge files, (music, images, binaries) the error margin would be very small. If you have a lot of small files (like scripts, sources and text files) then the error margin might easily add another 30 GB to the total file size, that you didn't account for...

So, measuring file sizes isn't easy...

share|improve this answer
    
Good point! , you could modify the stat statement (%b) in my answer and the number in the if statement to use block sizes instead. –  Kyle Brandt Jul 31 '09 at 10:36

You can get a listing of file usage either by pulling the size out of ls -l or by using the du command:

$ cd /dirwithlotsoffiles $ du -k *

That will print a list of the size of the files in kilobytes followed by the filename.

share|improve this answer
    
how will that help me add them together to move the first 300gb to one place and the second 300gb to a different place and so on? –  Bill Gray Jul 30 '09 at 15:53
    
As the other folks said, that is just the first step. Kyle provided a full solution, but what I had in mind was essentially to loop through the du output, adding up the sizes until you hit your threshold. –  Chad Huneycutt Jul 30 '09 at 20:42

The "find" answer would copy the first 300 files, not the first 300GB as I understand as the request.

You can try with tar and its multiple-volume options

share|improve this answer
    
I was thinking about that, but I'm not sure that tar wouldn't split a file between the two archives if that's where it got to when it got to the end of the fixed size. –  CK. Jul 30 '09 at 16:02

A pretty crude way would be to loop over files sorted by size (ls -S) and simply move each alternate file to one of the subdirectories. How about this:

#!/usr/bin/bash
dir1=path/to/dir1
dir2=path/to/dir2
a=0
for file in `ls -1S`
do
  a=`expr $a + 1`
  even=`expr $a%2|bc`
  if [ $even -gt 0 ]
  then
    mv $file $dir1
  else
    mv $file $dir2
  fi
done

~

share|improve this answer

I'm afraid you're probably going to have to get your hands dirty with some scripting here. You can easily get a list of files and their size using the terminal command ls -l, you'd then have to write a script that goes through that list and copies files one-by-one and keeps a counter to record the number of KB transferred so far. Each time check to see if we're moved 300GB's worth yet, if not, move another file. It's probably do-able in about 10 lines of Perl or less.

share|improve this answer

You can get a reasonable result by simply getting a list of filenames along with the size of each file. Sort the files according to size largest first. Then simply copy the largest file on the list that will fit in the remaining space on the target directory and remove it from the list. Repeat until no more files will fit.

Then start again with a new target directory. Repeat until the list is empty.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.