Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am getting memory related error when I try to split a big file into 8GB files.

# split -C 8000000000 r_mini_20120503.txt  
split: memory exhausted

I am however able to create 4GB files using the same command. Is there any way to increase memory for this purpose?

share|improve this question
2  
I doubt it's a memory issue, more likely the version of 'split' you are using doesn't support large files. Grab the src and recompile with '-D FILE_OFFSET_BITS=64' (this will work on recent 32bit as well as 64bit) –  symcbean May 14 '12 at 11:20
    
That's not the question though. –  Tom O'Connor May 14 '12 at 11:29

3 Answers 3

up vote 3 down vote accepted

Maybe split is the somewhat wrong tool for this task. You can try it with dd (untested):

infile=r_mini_20120503.txt
filesize=1234567890 # manual set
skip=0
chunk=8000000000
bs=1000000000
count=8
part=1

while [ $skip -lt $filesize ] ; do
    dd if=$infile of=$infile.$part bs=$bs count=$count skip=$skip
    skip=[[ $skip + $chunk ]]
    part=[[ $part + 1 ]]
done

As I wrote, untested.

share|improve this answer
1  
This is interesting, but deeply, deeply scary. –  Tom O'Connor May 14 '12 at 11:29
    
Why is that scary? –  ott-- May 14 '12 at 11:44
    
I find dd scary. –  Tom O'Connor May 14 '12 at 12:11
    
especially the "untested" part along with dd is scary :) –  the-wabbit May 14 '12 at 13:28

I'd say you're trying to use more memory than is on your system.
You could have a go at doing this in Single User mode, so there's less services, and other stuff (like X and GDM) running.

To do that, run telinit 1

OR.. Install more RAM. It's cheap.

share|improve this answer

How much RAM is in your box? You could add some more swap to your system for the time being, but that's going to be dog slow :P

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.