I need to use a command like the following:

$ cat large.input.file | process.py > large.output.file

The problem is, as this is won't the hard disk have a hard time jumping between reading the input file and writing the output file?

Is there a way to tell bash to use a large memory buffer when doing this kind of pipe?


Don't worry. The OS will do the buffering for you, and it's usually very good at it.

That being said: If you can change process.py you can implement your own buffering. If you can't change process.py you could write your own buffer.py and use it like this.

$ cat large.input.file | buffer.py | process.py | buffer.py > large.output.file

Probably much easier would be to read and write form a RAM disk.

  • Am I right to think that 'process.py' will already be buffered, automatically, by the Python interpreter? You don't need to implement your own buffering unless you want to override the buffer size (using 'python -u process.py' to turn off the Python interpreter IO buffer) May 17 '17 at 19:09

The OS will buffer the output to a certain amount, but there may still be a lot of head flipping if both the input and output files are on the same drive, unless your process.py does some buffering of its own.

You could replace cat in your example with pipe viewer (pv) (available in most standard repositories, and easily complied if it isn't in your distribution's repo) which allows you to set it to buffer more (with the -B/--buffer-bytes options) and displays a progress bar (unless you ask it not to) which could be very handy for a long operation if your process.py doesn't output its own progress information. For passing data from one place on a drive to an other place on the same drive this can make quite a difference unless the overall process is primarily CPU bound rather than I/O bound.

So for a 1Mb buffer you could do:

pv -B 1m large.input.file | process.py > large.output.file

I use pv all the time for this sort of thing, though mainly for the progress indicator more than the tweakable buffer size.

Another option is to use the more "standard" (standard in terms of being generally available by default, its command line format is a little different to most common commands) dd, though this does not have the progress bar facility:

dd if=large.input.file bs=1048576 | process.py > large.output.file

Edit: ps. pendants may point out that cat is not needed in your example as the following will work just as well and will be very slightly more efficient:

process.py < large.input.file > large.output.file

Some people refer to the removal of uneccassary calls to cat as "demogification", those these people should probably not be encouraged...

  • 3
    the trouble with "demogification" pedantry is that Useless Use Of Cat (UUOC) isn't actually useless. one very important use for it is in examples - it makes them much clearer and easier to read and understand. it's also convenient to test your pipeline with a small sample (using tail or head or grep or whatever) before running the entire huge file through it...then replace 'head' with 'cat' when the pipeline does exactly what you want.
    – cas
    Sep 5 '09 at 2:00
  • I would agree there. I often have an extra cat in place as it can improve the visual flow (for those of us who naturally read left-to-right) especially of a line that involves multiple redirections/pipes, making scripts slightly clearer and therefore a little easier to maintain. Sep 5 '09 at 15:38
  • If somebody truly thinks that running cat is too expensive, they should probably fix their shell to automatically replace cat _single_file_ | with < _single_file. I'll keep writing cat _single_file_ | instead. Mar 12 '19 at 11:09

Isn't there an old unix tool called "buffer"? Not that this would be needed with todays caching techniques - but it is there.

  • yep, there is. i use it often as a nice, flexible replacement for dd.
    – cas
    Sep 5 '09 at 1:56
  • Oh using as a dd replacement is a really a nice idea - it should be pretty much faster?
    – hurikhan77
    Sep 6 '09 at 12:54

I believe that the problem that the user is aluding to is how the Input/Output generally works in the UNIX/Linux world. Each UNIX/Linux process can basically have only one I/O operation pending at a time. Thus in the case of the cat command in the example, the cat command first reads some data, waits for it to complete, then writes the data and waits for it to complete before continuing. There is no concurrent I/O within a processes thus the buffering is used only between the reads and the writes to simply temporarily hold some data.

To speed things up, the input and output can be broken down over two different processes: one reader process and one writer process, and a lot of shared memory used as the buffer between the two processes. This results in the concurrent I/O that one desires and can speed up the file transfer process.

The utiilty program buffer, indicated by a user, implements this concurrent method that I described. I have used the buffer program with a fairly large shared memory buffer when interfacing with a tape drive for backups. This resulted in about a 20% decrease in wall clock transfer time.

Using the buffer program as a replacement for the 'cat' command might result in some definite improvements ... depending.



Try using this little Python 2 program I just put together:

#! /usr/bin/python2
# This executable path is Gentoo-specific, you might need to change it yourself

import sys;

if sys.argv[1].endswith('K'):
   bytestoread = int(sys.argv[1].translate(None, 'K')) * 1024;
elif sys.argv[1].endswith('M'):
   bytestoread = int(sys.argv[1].translate(None, 'M')) * 1024 * 1024;
elif sys.argv[1].endswith('G'):
   bytestoread = int(sys.argv[1].translate(None, 'G')) * 1024 * 1024 * 1024;

while True:
   buffer = sys.stdin.read(bytestoread);
   if buffer == '':
   buffer = None;   # This one is for making sure the read buffer will get destroyed, otherwise we could bring our system to a halt if we have 8 GB of RAM, request a 5 GB buffer, and it ends up eating 10 GB of memory.

To use this file, call it like this:

cat large.input.file | process.py | buffer.py 2G > large.output.file

You can use 2K to specify 2 kilobytes, 2M for 2 megabytes, 2G for 2 gigabytes, if you want to you might add 2T for 2 terabytes of buffer :3

I get this problem all the time when compressing a virtual machine image with pigz -1, because this makes the compression so incredibly fast the disk starts reading and writing simultaneously and the process slows to a grinding halt as the disk's head starts whizzing between the input and output files. So what I did was making this little program that reads a gargantuan block of data from standard input, writes it to standard output, and repeats. When the read returns a blank string, it's because no more standard input has been received and the script finishes.

  • At least in theory, read() can fail with E_AGAIN in UNIX-like systems and your program would then exit() prematurely. Always check for EOF explicitly. Mar 12 '19 at 11:14

It will buffer intelligently, but there's no good way to tweak how much that is.

You could write an intermediate program that would do the caching you want and have it read from the input.


Your OS will do all kinds of caching on files before they are written to the hard drive, and will also do caching on the file being read (generally reading ahead if possible). Let the OS do the buffering and caching.

Until you can prove using testing and profiling that the hard drive is the limiting factor in the equation it is best to just leave it alone.

If you can change process.py you could instead of reading/writing with pipes read/write directly and use buffering and or memmap'ed files instead which would help remove some of the load from the system.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.