Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a directory with 27K files, I would like to tar them into multiple INDEPENDENT tar files, each will have 5000 files and the last one, will obviously have 2K (27K is not dividable by 5).

What is the fastest/easiest way to do this?

share|improve this question

2 Answers 2

up vote 8 down vote accepted

First create files with the filenames for each archive:

find <directory> | split -l 5000 - files.

Then create the tars:

for f in files.*; do tar -cf $f.tar --files-from $f; done

Untested but the basic idea should work.

share|improve this answer
1  
This is probably the quickest (in terms of typing) / most portable (should work on any unix - find, split and for loops in Bourne shells are everywhere) solution :) –  voretaq7 Jan 26 '12 at 7:47
    
Use "find <directory> -type f" so that the pathnames passed into tar don't include directories. Alternative: for small split sizes, you could could use xargs insteads of split and avoid all the temp files. –  Liudvikas Bukys Jan 26 '12 at 14:25
#!/bin/bash
files=( dir/* )

n=1
for ((i = 0; i < ${#files[@]}; i+=5000)); do 
  IFS=$'\n' tar cvzf foo_$((n++)).tgz --files-from - <<<"${files[*]:i:5000}"
done

Explanation

Create an array called files that contains all contents of directory dir. Iterate over the array increasing the iterator 5000 at a time. Use the offset parameter expansion on the array to do a sliding window of up to 5000 items at a time "${files[@]:i:5000}" and pass that to tar and increment the file name by one $((n++)) on each call.

share|improve this answer
2  
Bash might throw an "argument list too long" at you if you do this. It's possible to pipe the filenames to tar if you use --files-from -. That might work better here. –  Rob Wouters Jan 26 '12 at 6:37
    
@RobWouters Great idea. Did it in a slightly different way but this shouldn't cause any max arg errors. –  SiegeX Jan 26 '12 at 6:59
1  
Aren't you passing all those arguments to printf now though? Same problem. –  Rob Wouters Jan 26 '12 at 7:04
    
@RobWouters doh, you're right! Ok, 3rd time's the charm –  SiegeX Jan 26 '12 at 7:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.