66

Looking for something like this? Any ideas?

cmd | prepend "[ERRORS] "

[ERROR] line1 text
[ERROR] line2 text
[ERROR] line3 text
... etc
3
  • 1
    Is there any way to set this up for all commands in bash function/script? May 8, 2018 at 3:43
  • What is cmd in this case?
    – Someone
    Aug 26, 2021 at 23:40
  • Does it matter? Assume is it 'ls'
    – user14645
    Aug 27, 2021 at 16:13

8 Answers 8

60
cmd | while read line; do echo "[ERROR] $line"; done

has the advantage of only using bash builtins so fewer processes will be created/destroyed so it should be a touch faster than awk or sed.

@tzrik points out that it might also make a nice bash function. Defining it like:

function prepend() { while read line; do echo "${1}${line}"; done; }

would allow it to be used like:

cmd | prepend "[ERROR] "
9
  • 4
    This actually only reduces the process count by one. (But it might be faster because no regexps (sed) or even string splitting (awk) are used.)
    – user1686
    Oct 9, 2009 at 7:52
  • 1
    BTW, I was curious about performance and here are results of my simple benchmark using bash, sed and awk. Pushing about 1000 lines of text (dmesg output) to FIFO file and then reading them like this: pastebin.ca/1606844 Looks like awk is the winner. Any ideas why? Oct 9, 2009 at 11:12
  • 2
    be careful running timing tests like that - try running them in all 6 different orders and then averaging the results. Different orders to mitigate block cache effects and average to mitigate background interruption/scheduling effects.
    – pjz
    Oct 9, 2009 at 18:25
  • 1
    Easy enough to wrap it in a function as well: function prepend() { while read line; do echo "${1}${line}"; done; }
    – tzrlk
    Feb 1, 2018 at 20:48
  • 1
    This works well, but it masks the exit code. Is there a way to do this while preserving the exit code? Apr 20, 2020 at 8:44
75

Try this:

cmd | awk '{print "[ERROR] " $0}'

Cheers

5
  • 1
    This has the disadvantage that "[ERROR]" cannot be a variable, because the whole expression must be in single-quotes. Nov 17, 2013 at 3:25
  • 4
    awk -vT="[ERROR] " '{ print T $0 }' or awk -vT="[ERROR]" '{ print T " " $0 }'
    – Tino
    Feb 21, 2014 at 23:30
  • 2
    T="[ERROR] " awk '{ print ENVIRON["T"] $0 }' or T="[ERROR]" awk '{ print ENVIRON["T"] " " $0 }'
    – Tino
    Feb 21, 2014 at 23:31
  • You can just exit the scope of the quotes to dereference the variable: cmd | awk '{print "['$V]' " $0}' - this should be evaluated once at the start, so no performance overhead.
    – robert
    Aug 7, 2018 at 9:10
  • 2
    I prefer this solution, as the bashism eats leading spaces, for instance the output of figlet. Jul 30, 2021 at 6:24
27

With all due credit to @grawity, I'm submitting his comment as an answer, as it seems the best answer here to me.

sed 's/^/[ERROR] /' cmd
5
  • Why is this preferable to the bash solution?
    – user14645
    Jan 31, 2012 at 18:04
  • 3
    I suppose it depends on your purpose. If your goal is to simply prepend to every line in a file, this accomplishes that goal with very few characters, using a very familiar tool. I far prefer that to a 10 line bash script. The awk one-liner is nice enough, but I think that more people are familiar with sed than awk. The bash script is good for what it does, but it seems that it is answering a question that was not asked. Jan 31, 2012 at 18:22
  • The answer that pjz submitted is also a nice one-liner. It doesn't additional programs, processes and may run a little quicker.
    – user14645
    Feb 2, 2012 at 18:11
  • 5
    sed X cmd does read cmd and does not execute it. Either cmd | sed 's/^/[ERROR] /' or sed 's/^/[ERROR] /' <(cmd) or cmd > >(sed 's/^/[ERROR] /'). But beware the latter. Even that this allows you to access the return value of cmd the sed runs in background, so it is likely you see the output after cmd finished. Good for logging into a file, though. And note that awk probably is faster than sed.
    – Tino
    Feb 21, 2014 at 23:41
  • Nice. This command is easily aliased. alias lpad="sed 's/^/ /'". instead of ERROR I insert 4 leading spaces. Now, for the magic trick: ls | lpad | pbcopy will prepend ls output with 4 spaces which marks it as Markdown for code, meaning that you paste the clipboard (pbcopy grabs it, on macs) directly into StackOverflow or any other markdown context. Couldn't alias the awk answer (on 1st try) so this one wins. The while read solution is also alias-able, but I find this sed more expressive.
    – JL Peyret
    Dec 8, 2017 at 19:59
13

I created a GitHub repository to do some speed tests.

The result is:

  • In the general case, awk is fastest. sed is a bit slower and perl is not much slower than sed. Apparently, all those are highly optimized languages for text processing.
  • In very special situations, where forks dominate, running your script as a compiled ksh script (shcomp) can save even more processing time. In contrast, bash is dead slow compared to compiled ksh scripts.
  • Creating a statically linked binary to beat awk seems not be worth the effort.

In contrast python is dead slow, but I have not tested a compiled case, because it is usually not what you would do in such a scripting case.

Following variants are tested:

while read line; do echo "[TEST] $line"; done
while read -r line; do echo "[TEST] $line"; done
while read -r line; do echo "[TEST]" $line; done
while read -r line; do echo "[TEST]" "$line"; done
sed 's/^/[TEST] /'
awk '{ print "[TEST] " $0 }'
awk -vT="[TEST] " '{ print T $0 }'
awk -vT="[TEST]" '{ print T " " $0 }'
awk -vT="[TEST]" 'BEGIN { T=T " "; } { print T $0 }'
T="[TEST] " awk '{ print ENVIRON["T"] $0 }'
T="[TEST]" awk '{ print ENVIRON["T"] " " $0 }'
T="[TEST]" awk 'BEGIN { T=ENVIRON["T"] " " } { print T $0 }'
perl -ne 'print "[TEST] $_"'

Two binary variants of one of my tools (it is not optimzed for speed, though):

./unbuffered.dynamic -cp'[TEST] ' -q ''
./unbuffered.static -cp'[TEST] ' -q ''

Python buffered:

python -uSc 'import sys
for line in sys.stdin: print "[TEST]",line,'

And Python unbuffered:

python -uSc 'import sys
while 1:
 line = sys.stdin.readline()
 if not line: break
 print "[TEST]",line,'
1
  • awk -v T="[TEST %Y%m%d-%H%M%S] " '{ print strftime(T) $0 }' to output a timestamp
    – Tino
    Feb 22, 2014 at 4:58
5
cmd | sed 's/.*/[ERROR] &/'
1
  • 16
    sed 's/^/[ERROR] /'
    – user1686
    Oct 9, 2009 at 7:49
4

I wanted a solution that handled stdout and stderr, so I wrote prepend.sh and put it in my path:

#!/bin/bash

prepend_lines(){
  local prepended=$1
  while read line; do
    echo "$prepended" "$line"
  done
}

tag=$1

shift

"$@" > >(prepend_lines "$tag") 2> >(prepend_lines "$tag" 1>&2)

Now I can just run prepend.sh "[ERROR]" cmd ..., to prepend "[ERROR]" to the output of cmd, and still have stderr and stdout separate.

1
  • I tried this approach but there was something going on with those >( subshells that I couldn't quite resolve. It seemed as though the script was completing, and the output was arriving to the terminal after the prompt had returned which was a bit messy. I eventually came up with the answer here stackoverflow.com/a/25948606/409638
    – robert
    Sep 20, 2014 at 12:20
3
cmd | xargs -L 1 -i echo "prefix{}"

or even easier in case prefix is space-delimited from the line itself

cmd | xargs -L 1 echo prefix

This is not very efficient performance-wise, but short to write.

It works by running echo once per each line of input. xargs allows you to also process \0-delimited lines.

1
  • This has the benefit of working on windows natively if you have xargs in your path - like in git or mingw32. Thanks!
    – dgo
    May 11 at 13:40
0

Moreutils has the ts command, which can be abused to do what you need:

cmd | ts "[ERRORS]"

It's probably slower than the other solutions, but much shorter...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.