I have a directory with 27K files, I would like to tar them into multiple INDEPENDENT tar files, each will have 5000 files and the last one, will obviously have 2K (27K is not dividable by 5).

What is the fastest/easiest way to do this?


First create files with the filenames for each archive:

find <directory> | split -l 5000 - files.

Then create the tars:

for f in files.*; do tar -cf $f.tar --files-from $f; done

Untested but the basic idea should work.

  • 1
    This is probably the quickest (in terms of typing) / most portable (should work on any unix - find, split and for loops in Bourne shells are everywhere) solution :) – voretaq7 Jan 26 '12 at 7:47
  • Use "find <directory> -type f" so that the pathnames passed into tar don't include directories. Alternative: for small split sizes, you could could use xargs insteads of split and avoid all the temp files. – Liudvikas Bukys Jan 26 '12 at 14:25
files=( dir/* )

for ((i = 0; i < ${#files[@]}; i+=5000)); do 
  IFS=$'\n' tar cvzf foo_$((n++)).tgz --files-from - <<<"${files[*]:i:5000}"


Create an array called files that contains all contents of directory dir. Iterate over the array increasing the iterator 5000 at a time. Use the offset parameter expansion on the array to do a sliding window of up to 5000 items at a time "${files[@]:i:5000}" and pass that to tar and increment the file name by one $((n++)) on each call.

  • 2
    Bash might throw an "argument list too long" at you if you do this. It's possible to pipe the filenames to tar if you use --files-from -. That might work better here. – Rob Wouters Jan 26 '12 at 6:37
  • @RobWouters Great idea. Did it in a slightly different way but this shouldn't cause any max arg errors. – SiegeX Jan 26 '12 at 6:59
  • 1
    Aren't you passing all those arguments to printf now though? Same problem. – Rob Wouters Jan 26 '12 at 7:04
  • @RobWouters doh, you're right! Ok, 3rd time's the charm – SiegeX Jan 26 '12 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.