4

I have a directory with 27K files, I would like to tar them into multiple INDEPENDENT tar files, each will have 5000 files and the last one, will obviously have 2K (27K is not dividable by 5).

What is the fastest/easiest way to do this?

2 Answers 2

9

First create files with the filenames for each archive:

find <directory> | split -l 5000 - files.

Then create the tars:

for f in files.*; do tar -cf $f.tar --files-from $f; done

Untested but the basic idea should work.

2
  • 1
    This is probably the quickest (in terms of typing) / most portable (should work on any unix - find, split and for loops in Bourne shells are everywhere) solution :)
    – voretaq7
    Jan 26, 2012 at 7:47
  • Use "find <directory> -type f" so that the pathnames passed into tar don't include directories. Alternative: for small split sizes, you could could use xargs insteads of split and avoid all the temp files. Jan 26, 2012 at 14:25
0
#!/bin/bash
files=( dir/* )

n=1
for ((i = 0; i < ${#files[@]}; i+=5000)); do 
  IFS=$'\n' tar cvzf foo_$((n++)).tgz --files-from - <<<"${files[*]:i:5000}"
done

Explanation

Create an array called files that contains all contents of directory dir. Iterate over the array increasing the iterator 5000 at a time. Use the offset parameter expansion on the array to do a sliding window of up to 5000 items at a time "${files[@]:i:5000}" and pass that to tar and increment the file name by one $((n++)) on each call.

4
  • 2
    Bash might throw an "argument list too long" at you if you do this. It's possible to pipe the filenames to tar if you use --files-from -. That might work better here. Jan 26, 2012 at 6:37
  • @RobWouters Great idea. Did it in a slightly different way but this shouldn't cause any max arg errors.
    – SiegeX
    Jan 26, 2012 at 6:59
  • 1
    Aren't you passing all those arguments to printf now though? Same problem. Jan 26, 2012 at 7:04
  • @RobWouters doh, you're right! Ok, 3rd time's the charm
    – SiegeX
    Jan 26, 2012 at 7:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .