Is there a way to redirect output to a file without buffering on unix/linux?

Question

I have a long running batch process that outputs some debug and process information to stdout. If I just run from a terminal I can keep track of 'where it is' but then the data gets too much and scrolls off the screen.

If I redirect to output to a file '> out.txt' I get the whole output eventually but it is buffered so I can no longer see what it is doing right now.

Is there a way to redirect the output but make it not buffer its writes?

Answer 1

You can explicitly set the buffering options of the standard streams using a setvbuf call in C (see this link), but if you're trying to modify the behaviour of an existing program try stdbuf (part of coreutils starting with version 7.5 apparently).

This buffers stdout up to a line:

stdbuf -oL command > output

This disables stdout buffering altogether:

stdbuf -o0 command > output

Answer 2

You may achieve line buffered output to a file by using the script command like so:

stty -echo -onlcr   # avoid added \r in output
script -q /dev/null batch_process | tee output.log        # Mac OS X, FreeBSD
script -q -c "batch_process" /dev/null | tee output.log   # Linux
stty echo onlcr

Answer 3

On Ubuntu, the unbuffer program (from the expect-dev) package did the trick for me. Just run:

unbuffer your_command

and it won't buffer it.

Answer 4

The easiest solution that I found (didn't need any third-party packages installed) was mentioned in a similar thread over on at Unix & Linux site: use the script command. It's old, and likely already installed.

$ script -q /dev/null long_running_command | print_progress       # FreeBSD, Mac OS X
$ script -q -c "long_running_command" /dev/null | print_progress  # Linux

Note that the first filename parameter for the script command is the log file to be written. If you simply run script -q your_command, you'll overwrite the command you indented to run with the log file. Check man script, to be safe, before trying it.

Answer 5

try the script command; if your system has it, it takes a file name as argument, all text dumped to stdout gets copied to the file. It's very useful when a setup program requires interaction.

Answer 6

Personally I prefer piping output of a command I want to examine through tee.

script records too much information, including timing of key presses, and a lot of non-printable characters. What tee saves is much more human readable for me.

Answer 7

Redirect the output into a file and follow the file with the tail -f command.

Edit

If this still suffers from buffering, then use the syslog facility (which is generally unbuffered). If the batch process runs as a shell script, you can use the logger command to do this. If the batch job runs in a scripting language, there should be a logging facility anyway.

Answer 8

You can use the tee command, just magic !

someCommand | tee logFile.log will both display in the console and write into the log file.