35

Looking for something like this? Any ideas?

cmd | prepend "[ERRORS] "

[ERROR] line1 text
[ERROR] line2 text
[ERROR] line3 text
... etc
  • Is there any way to set this up for all commands in bash function/script? – Alexander Mills May 8 '18 at 3:43
38
cmd | while read line; do echo "[ERROR] $line"; done

has the advantage of only using bash builtins so fewer processes will be created/destroyed so it should be a touch faster than awk or sed.

@tzrik points out that it might also make a nice bash function. Defining it like:

function prepend() { while read line; do echo "${1}${line}"; done; }

would allow it to be used like:

cmd | prepend "[ERROR] "
  • 4
    This actually only reduces the process count by one. (But it might be faster because no regexps (sed) or even string splitting (awk) are used.) – grawity Oct 9 '09 at 7:52
  • BTW, I was curious about performance and here are results of my simple benchmark using bash, sed and awk. Pushing about 1000 lines of text (dmesg output) to FIFO file and then reading them like this: pastebin.ca/1606844 Looks like awk is the winner. Any ideas why? – Ilya Zakreuski Oct 9 '09 at 11:12
  • 1
    be careful running timing tests like that - try running them in all 6 different orders and then averaging the results. Different orders to mitigate block cache effects and average to mitigate background interruption/scheduling effects. – pjz Oct 9 '09 at 18:25
  • This question is tagged "shell", not "bash". – fiatjaf Jun 24 '16 at 15:12
  • 1
    Easy enough to wrap it in a function as well: function prepend() { while read line; do echo "${1}${line}"; done; } – tzrlk Feb 1 '18 at 20:48
44

Try this:

cmd | awk '{print "[ERROR] " $0}'

Cheers

  • 1
    This has the disadvantage that "[ERROR]" cannot be a variable, because the whole expression must be in single-quotes. – user1071136 Nov 17 '13 at 3:25
  • 4
    awk -vT="[ERROR] " '{ print T $0 }' or awk -vT="[ERROR]" '{ print T " " $0 }' – Tino Feb 21 '14 at 23:30
  • 2
    T="[ERROR] " awk '{ print ENVIRON["T"] $0 }' or T="[ERROR]" awk '{ print ENVIRON["T"] " " $0 }' – Tino Feb 21 '14 at 23:31
  • You can just exit the scope of the quotes to dereference the variable: cmd | awk '{print "['$V]' " $0}' - this should be evaluated once at the start, so no performance overhead. – robert Aug 7 '18 at 9:10
12

With all due credit to @grawity, I'm submitting his comment as an answer, as it seems the best answer here to me.

sed 's/^/[ERROR] /' cmd
  • Why is this preferable to the bash solution? – user14645 Jan 31 '12 at 18:04
  • 1
    I suppose it depends on your purpose. If your goal is to simply prepend to every line in a file, this accomplishes that goal with very few characters, using a very familiar tool. I far prefer that to a 10 line bash script. The awk one-liner is nice enough, but I think that more people are familiar with sed than awk. The bash script is good for what it does, but it seems that it is answering a question that was not asked. – Eric Wilson Jan 31 '12 at 18:22
  • The answer that pjz submitted is also a nice one-liner. It doesn't additional programs, processes and may run a little quicker. – user14645 Feb 2 '12 at 18:11
  • 3
    sed X cmd does read cmd and does not execute it. Either cmd | sed 's/^/[ERROR] /' or sed 's/^/[ERROR] /' <(cmd) or cmd > >(sed 's/^/[ERROR] /'). But beware the latter. Even that this allows you to access the return value of cmd the sed runs in background, so it is likely you see the output after cmd finished. Good for logging into a file, though. And note that awk probably is faster than sed. – Tino Feb 21 '14 at 23:41
  • Nice. This command is easily aliased. alias lpad="sed 's/^/ /'". instead of ERROR I insert 4 leading spaces. Now, for the magic trick: ls | lpad | pbcopy will prepend ls output with 4 spaces which marks it as Markdown for code, meaning that you paste the clipboard (pbcopy grabs it, on macs) directly into StackOverflow or any other markdown context. Couldn't alias the awk answer (on 1st try) so this one wins. The while read solution is also alias-able, but I find this sed more expressive. – JL Peyret Dec 8 '17 at 19:59
8

I created a GitHub repository to do some speed tests.

The result is:

  • In the general case, awk is fastest. sed is a bit slower and perl is not much slower than sed. Apparently, all those are highly optimized languages for text processing.
  • In very special situations, where forks dominate, running your script as a compiled ksh script (shcomp) can save even more processing time. In contrast, bash is dead slow compared to compiled ksh scripts.
  • Creating a statically linked binary to beat awk seems not be worth the effort.

In contrast python is dead slow, but I have not tested a compiled case, because it is usually not what you would do in such a scripting case.

Following variants are tested:

while read line; do echo "[TEST] $line"; done
while read -r line; do echo "[TEST] $line"; done
while read -r line; do echo "[TEST]" $line; done
while read -r line; do echo "[TEST]" "$line"; done
sed 's/^/[TEST] /'
awk '{ print "[TEST] " $0 }'
awk -vT="[TEST] " '{ print T $0 }'
awk -vT="[TEST]" '{ print T " " $0 }'
awk -vT="[TEST]" 'BEGIN { T=T " "; } { print T $0 }'
T="[TEST] " awk '{ print ENVIRON["T"] $0 }'
T="[TEST]" awk '{ print ENVIRON["T"] " " $0 }'
T="[TEST]" awk 'BEGIN { T=ENVIRON["T"] " " } { print T $0 }'
perl -ne 'print "[TEST] $_"'

Two binary variants of one of my tools (it is not optimzed for speed, though):

./unbuffered.dynamic -cp'[TEST] ' -q ''
./unbuffered.static -cp'[TEST] ' -q ''

Python buffered:

python -uSc 'import sys
for line in sys.stdin: print "[TEST]",line,'

And Python unbuffered:

python -uSc 'import sys
while 1:
 line = sys.stdin.readline()
 if not line: break
 print "[TEST]",line,'
  • awk -v T="[TEST %Y%m%d-%H%M%S] " '{ print strftime(T) $0 }' to output a timestamp – Tino Feb 22 '14 at 4:58
5
cmd | sed 's/.*/[ERROR] &/'
  • 16
    sed 's/^/[ERROR] /' – grawity Oct 9 '09 at 7:49
3

I wanted a solution that handled stdout and stderr, so I wrote prepend.sh and put it in my path:

#!/bin/bash

prepend_lines(){
  local prepended=$1
  while read line; do
    echo "$prepended" "$line"
  done
}

tag=$1

shift

"$@" > >(prepend_lines "$tag") 2> >(prepend_lines "$tag" 1>&2)

Now I can just run prepend.sh "[ERROR]" cmd ..., to prepend "[ERROR]" to the output of cmd, and still have stderr and stdout separate.

  • I tried this approach but there was something going on with those >( subshells that I couldn't quite resolve. It seemed as though the script was completing, and the output was arriving to the terminal after the prompt had returned which was a bit messy. I eventually came up with the answer here stackoverflow.com/a/25948606/409638 – robert Sep 20 '14 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.