1

I have a folder full of 600gb of files. I want to automatically copy the first 300 to one folder, and the rest to another folder. I am not sure how to limit the results with ls or whichever so I can pass it as an argument...

platform is linux...

edit: I want to move 300gb, not the first 300 files. File sizes are arbitrary, and ordering does not matter.

5
  • "first 300"? 300 files, or 300GB? Commented Jul 30, 2009 at 15:49
  • It seems, first 300gb, if i get the english right.
    – nik
    Commented Jul 30, 2009 at 15:51
  • 1
    Are you wanting to move 300 files (as in the answer below) or 1/2 of the size of files? If so: - How many files are there? - Is the size distribution even across the files?
    – CK.
    Commented Jul 30, 2009 at 15:51
  • 1
    Oh, and what do you mean by 'first'? Is there some ordering you want, or do you want to just split the directory into two smaller ones?
    – CK.
    Commented Jul 30, 2009 at 15:53
  • first 300gb, sorry. file sizes are aritrary
    – Bill Gray
    Commented Jul 30, 2009 at 15:54

9 Answers 9

5

Update: Oh, the first 300GB, well then... this probably slow, depending on file size, but I like the exercise :-)

filesize=0
for i in *; do 
    filesize=$(stat -c "%s" "$i");  
    (( totalsize += filesize )); 
    if [[ $totalsize < 322122547200 ]]; then
        mv "$i" first_300/
    else
        mv "$i" the_rest/
    fi
done

Hopefully there are not problems with the size of the int.


If you mean break them up into folders each with 300 files, maybe you want something like the following:

folder=0
counter=0
for i in *; do 
    mv $i foo_$folder/
    if [[ $(( counter % 10 )) -eq 0 ]]; then 
       (( folder++ ));
    fi
    (( counter++ ))
done

Although that might not be as fast as some of the find commands. If you just want to do the first 300 command, you could use the same counter strategy but use a while $counter -le 300.

1
  • 1
    For fairly current bash shells integer math should be able to handle (2^64)-1 without wrapping around. If sizing is a problem then you can use du -k $i | awk '{print $1} in place of the stat statement. If you still want to use stat you may want to try counting the blocks used instead of bytes. This would mean using %b in place of $s as the format argument. 'stat -c "%B" /mountpoint', will tell you your blocksize for scaling. Commented Jul 30, 2009 at 16:48
2

This is a way to get nearly equal 300gb distribution,

You could do a du based search to find distribution across top level directories and files and then split them into nearly two parts with some trials.

find . -maxdepth 1 -type d -exec du -sk {} \; | sort -n -k 1 > list.txt

This will give a sorted list of KB sizes.
You could do a trick like picking up alternate lines on this list for a quick nearly-even distribution

awk '{if (FNR%2==1) print $2}' list.txt > list1.txt
awk '{if (FNR%2==0) print $2}' list.txt > list2.txt

A Very rough distribution...

Finally, if you have very uneven file or directory sizes -- quite far from 300GB distribution,
keep yourself away from the bin-packing problem and do some simple trials on moving around a couple of lines between the two list files.
Find the difference between the two sets (with du) and move a directory/file
that is about half the difference from the larger list to the smaller one.
That should get you quite close

4
  • 3
    Anything other than rough would mean solving discrete knapsack, so let's hope... :) Commented Jul 30, 2009 at 16:37
  • @Anders Eurenus - +1 exactly what I was thinking. "Congratulations, you've just invented the Knapsack problem." Fortunately, 300GB is probably just an approximation or an upper limit, so a greedy heuristic is probably fine. Commented Jul 30, 2009 at 16:43
  • This is actually a bin packing problem with two bins. I think, Kyles answer can be tweaked to use a list sorted for size (largest first) to give quite an optimal solution. This answer tries something similar with the sorting. It is probably simpler and slightly less efficient.
    – nik
    Commented Jul 31, 2009 at 2:31
  • I think, it is less efficient than a tweaked version of Kyles answer. But, probably better than his present unsorted version.
    – nik
    Commented Jul 31, 2009 at 2:32
1

You could do it with find, head & xargs. It should look like this:

find ./ -type f -print0 | head -300 | xargs -0 -I mv {} /one/folder
find ./ -type f -print0 | xargs -0 -I mv {} /another/folder
5
  • 2
    Was this tried on a shell in any way?
    – nik
    Commented Jul 30, 2009 at 15:50
  • 1
    You may have to do it this way: ./ -type f -print0 | head -300 | xargs -0 -I myvar cp myvar ./destfolder/
    – Maxwell
    Commented Jul 30, 2009 at 16:17
  • @Nik: Not at all. I only have AIX machine right now. This is why I said "should".
    – Benoit
    Commented Jul 30, 2009 at 17:10
  • What's your reasoning for using "find", aside from the fact that it will search the directory tree? The OP says that the files are all in one folder, so I would expect that "ls" would be a faster way. Since the OP was trying to divide up the directory by size, "ls -l |sort -n -k 5" would be a good starting place.
    – Ernie
    Commented Jul 30, 2009 at 18:16
  • @Ernie: It has been said that the folder contains all the files to move not that it contains /only/ these files...
    – Benoit
    Commented Jul 31, 2009 at 7:48
1

WARNING! When you start calculating file sizes, you are likely to make the mistake to measure them by bytes, while most file systems will allocate disk space in blocks. And this block size varies from disk to disk but is often a multiple of 512.

Basically, that means you can have 500 files of one byte each, which would only be 500 bytes. But a file-system that allocates 2048 bytes per block would thus claim about 1 megabyte of disk space. Yeah, that's a lot of overhead.

Basically, you should round up the file-sizes you get by the block size of the file system you use. That way, you can measure them more precisely.

Then again, how much difference could it be? If the block size is 2048 bytes then the average amount of bytes "lost" would be 1 KB. With 300 files this would be about 300 KB that you would need more on top of your total size. You want to copy 300 GB but how many files would that be? And are the two disks using the same file system with the same block size?

Anyway, the error margin depends on the average file-size. If you have a lot of huge files, (music, images, binaries) the error margin would be very small. If you have a lot of small files (like scripts, sources and text files) then the error margin might easily add another 30 GB to the total file size, that you didn't account for...

So, measuring file sizes isn't easy...

1
  • Good point! , you could modify the stat statement (%b) in my answer and the number in the if statement to use block sizes instead. Commented Jul 31, 2009 at 10:36
0

You can get a listing of file usage either by pulling the size out of ls -l or by using the du command:

$ cd /dirwithlotsoffiles $ du -k *

That will print a list of the size of the files in kilobytes followed by the filename.

2
  • how will that help me add them together to move the first 300gb to one place and the second 300gb to a different place and so on?
    – Bill Gray
    Commented Jul 30, 2009 at 15:53
  • As the other folks said, that is just the first step. Kyle provided a full solution, but what I had in mind was essentially to loop through the du output, adding up the sizes until you hit your threshold. Commented Jul 30, 2009 at 20:42
0

The "find" answer would copy the first 300 files, not the first 300GB as I understand as the request.

You can try with tar and its multiple-volume options

1
  • I was thinking about that, but I'm not sure that tar wouldn't split a file between the two archives if that's where it got to when it got to the end of the fixed size.
    – CK.
    Commented Jul 30, 2009 at 16:02
0

A pretty crude way would be to loop over files sorted by size (ls -S) and simply move each alternate file to one of the subdirectories. How about this:

#!/usr/bin/bash
dir1=path/to/dir1
dir2=path/to/dir2
a=0
for file in `ls -1S`
do
  a=`expr $a + 1`
  even=`expr $a%2|bc`
  if [ $even -gt 0 ]
  then
    mv $file $dir1
  else
    mv $file $dir2
  fi
done

~

0

I'm afraid you're probably going to have to get your hands dirty with some scripting here. You can easily get a list of files and their size using the terminal command ls -l, you'd then have to write a script that goes through that list and copies files one-by-one and keeps a counter to record the number of KB transferred so far. Each time check to see if we're moved 300GB's worth yet, if not, move another file. It's probably do-able in about 10 lines of Perl or less.

0

You can get a reasonable result by simply getting a list of filenames along with the size of each file. Sort the files according to size largest first. Then simply copy the largest file on the list that will fit in the remaining space on the target directory and remove it from the list. Repeat until no more files will fit.

Then start again with a new target directory. Repeat until the list is empty.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .