71

I have a long running batch process that outputs some debug and process information to stdout. If I just run from a terminal I can keep track of 'where it is' but then the data gets too much and scrolls off the screen.

If I redirect to output to a file '> out.txt' I get the whole output eventually but it is buffered so I can no longer see what it is doing right now.

Is there a way to redirect the output but make it not buffer its writes?

2
  • 1
    Could you please have a look at my (and @cnst 's) "debate" below, I'm guessing the only thing you want is to see the output in the same time as logging it to a file. If you found a solution, let us know about it ;) !
    – Benj
    Commented Dec 2, 2013 at 16:22
  • 2
    more upvoted question unix.stackexchange.com/questions/25372 Commented Jun 10, 2016 at 20:20

8 Answers 8

71

You can explicitly set the buffering options of the standard streams using a setvbuf call in C (see this link), but if you're trying to modify the behaviour of an existing program try stdbuf (part of coreutils starting with version 7.5 apparently).

This buffers stdout up to a line:

stdbuf -oL command > output

This disables stdout buffering altogether:

stdbuf -o0 command > output
5
  • aaarghhh... My Ubuntu has only 7.4 coreutils... :(
    – Calmarius
    Commented Feb 18, 2013 at 11:38
  • @Calmarius: compiling coreutils should be quite easy. Just grab a new version from ftp.gnu.org/gnu/coreutils and give it a go. It's standard ./configure && make fare. You don't even have to install it afterwards, you can just use the stdbuf binary off src/. Commented Feb 18, 2013 at 14:52
  • double argh. i need this on an old centos distro AND OSX as well as Ubuntu.
    – edk750
    Commented Apr 16, 2014 at 0:00
  • more upvoted answer unix.stackexchange.com/a/25378/5510 Commented Jun 10, 2016 at 20:21
  • Is there a way to force the pipe/printf buffers externally for already running process with known PID?
    – mvorisek
    Commented Jun 9, 2019 at 22:31
11

You may achieve line buffered output to a file by using the script command like so:

stty -echo -onlcr   # avoid added \r in output
script -q /dev/null batch_process | tee output.log        # Mac OS X, FreeBSD
script -q -c "batch_process" /dev/null | tee output.log   # Linux
stty echo onlcr
1
  • 1
    -1 because: a) unlike the accepted answer this doesn't work if you want to run the command in the background (the command immediately terminates without finishing batch_process if you append & to the command above, at least on my Linux box), which seems like an extremely common use case, and b) there's no explanation here of how this incantation works.
    – Mark Amery
    Commented Oct 10, 2016 at 10:16
10

On Ubuntu, the unbuffer program (from the expect-dev) package did the trick for me. Just run:

unbuffer your_command

and it won't buffer it.

6

The easiest solution that I found (didn't need any third-party packages installed) was mentioned in a similar thread over on at Unix & Linux site: use the script command. It's old, and likely already installed.

$ script -q /dev/null long_running_command | print_progress       # FreeBSD, Mac OS X
$ script -q -c "long_running_command" /dev/null | print_progress  # Linux

Note that the first filename parameter for the script command is the log file to be written. If you simply run script -q your_command, you'll overwrite the command you indented to run with the log file. Check man script, to be safe, before trying it.

4

try the script command; if your system has it, it takes a file name as argument, all text dumped to stdout gets copied to the file. It's very useful when a setup program requires interaction.

1
  • I know of the 'script -a out.txt' trick. I was wondering if there is any other way to make the writing process not buffer.
    – James Dean
    Commented Jul 27, 2011 at 14:56
3

Personally I prefer piping output of a command I want to examine through tee.

script records too much information, including timing of key presses, and a lot of non-printable characters. What tee saves is much more human readable for me.

2
  • I would also add " | less" to the command line.
    – HUB
    Commented Jul 26, 2011 at 15:43
  • 5
    I'm pretty sure that tee is affected by buffering also. I frequently still get partial lines displayed by tee when splitting output from find commands.
    – Magellan
    Commented Jul 26, 2011 at 16:19
2

Redirect the output into a file and follow the file with the tail -f command.

Edit

If this still suffers from buffering, then use the syslog facility (which is generally unbuffered). If the batch process runs as a shell script, you can use the logger command to do this. If the batch job runs in a scripting language, there should be a logging facility anyway.

3
  • 3
    This is what I do right now but the process buffers the write to the file and that is just what I want to avoid.
    – James Dean
    Commented Jul 27, 2011 at 14:54
  • See my edit for an updated proposal.
    – wolfgangsz
    Commented Jul 27, 2011 at 15:12
  • 2
    This doesn't work for obvious reasons. Who the hell gives out This answer is useful for answers that don't work, and can't possibly work?
    – cnst
    Commented Nov 24, 2013 at 9:14
2

You can use the tee command, just magic !

someCommand | tee logFile.log will both display in the console and write into the log file.

4
  • 2
    Doesn't work. tee will not stop any buffering from taking place.
    – cnst
    Commented Nov 24, 2013 at 9:15
  • 1
    @cnst Effectively, tee will not avoid some buffering but will only allow you to have a look on what is the output. This is what @JamesDean wanted (as I undestood his question), but I think buffering is not really the problem here. If you have more details, let me know.
    – Benj
    Commented Nov 30, 2013 at 10:14
  • He wanted to see output, and he's not getting any output (in an unbuffered manner), and yet you suggest to use tee to get a better view at the output that one isn't getting?
    – cnst
    Commented Nov 30, 2013 at 17:56
  • 1
    How I red @JamesDean question : "I redirect to output to a file '> out.txt'" means for me no console output, wait for process to complete while the entire output is redirected. When you are using > you don't see anything on the console. I guess @JamesDean is using the word "buffering" to describe that. I'll post a comment on his question for him to say more about what he wants.
    – Benj
    Commented Dec 2, 2013 at 16:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .