11

I've a 100GB file and I want to split into 100 of 1GB file each (by line break)

e.g.

split --bytes=1024M /path/to/input /path/to/output

For the 100 files generated, I want to apply gzip/zip to each of these files.

Is it possible to use a single command?

  • 2
    For up to 1GB per file (less if the next line would put it over) use --line-bytes=1024M. – Brian May 26 '14 at 19:04
28

Use "--filter":

split --bytes=1024M --filter='gzip > $FILE.gz' /path/to/input /path/to/output

0

A one-liner using a conditional is as close as you can come.

cd /path/to/output && split --bytes=1024M /path/to/input/filename && gzip x*

gzip will only run if split is successful because of the conditional && which is also between the cd and split making sure the cd is successful, too.. Note that split and gzip output to the current directory instead of having the ability to specify the output directory. You can make the directory, if needed:

mkdir -p /path/to/output && cd /path/to/output && split --bytes=1024M /path/to/input/filename && gzip x*

To put it all back together:

gunzip /path/to/files/x* && cat /path/to/files/x* > /path/to/dest/filename
0

Using this command with -d option allows you to generate numeric sufixes.

split -d -b 2048m "myDump.dmp" "myDump.dmp.part-" && gzip myDump.dmp.part*

Files generated:

    myDump.dmp.part-00
    myDump.dmp.part-01
    myDump.dmp.part-02
    ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.