Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Looking for something like this? Any ideas?

cmd | prepend "[ERRORS] "

[ERROR] line1 text
[ERROR] line2 text
[ERROR] line3 text
... etc
share|improve this question

6 Answers 6

up vote 13 down vote accepted
cmd | while read line; do echo "[ERROR] $line"; done

has the advantage of only using bash builtins so fewer processes will be created/destroyed so it should be a touch faster than awk or sed.

share|improve this answer
2  
This actually only reduces the process count by one. (But it might be faster because no regexps (sed) or even string splitting (awk) are used.) –  grawity Oct 9 '09 at 7:52
    
BTW, I was curious about performance and here are results of my simple benchmark using bash, sed and awk. Pushing about 1000 lines of text (dmesg output) to FIFO file and then reading them like this: pastebin.ca/1606844 Looks like awk is the winner. Any ideas why? –  Ilya Zakreuski Oct 9 '09 at 11:12
1  
be careful running timing tests like that - try running them in all 6 different orders and then averaging the results. Different orders to mitigate block cache effects and average to mitigate background interruption/scheduling effects. –  pjz Oct 9 '09 at 18:25

Try this:

cmd | awk '{print "[ERROR] " $0}'

Cheers

share|improve this answer
    
This has the disadvantage that "[ERROR]" cannot be a variable, because the whole expression must be in single-quotes. –  user1071136 Nov 17 '13 at 3:25
    
awk -vT="[ERROR] " '{ print T $0 }' or awk -vT="[ERROR]" '{ print T " " $0 }' –  Tino Feb 21 at 23:30
    
T="[ERROR] " awk '{ print ENVIRON["T"] $0 }' or T="[ERROR]" awk '{ print ENVIRON["T"] " " $0 }' –  Tino Feb 21 at 23:31
cmd | sed 's/.*/[ERROR] &/'
share|improve this answer
7  
sed 's/^/[ERROR] /' –  grawity Oct 9 '09 at 7:49

I wanted a solution that handled stdout and stderr, so I wrote prepend.sh and put it in my path:

#!/bin/bash

prepend_lines(){
  local prepended=$1
  while read line; do
    echo "$prepended" "$line"
  done
}

tag=$1

shift

"$@" > >(prepend_lines "$tag") 2> >(prepend_lines "$tag" 1>&2)

Now I can just run prepend.sh "[ERROR]" cmd ..., to prepend "[ERROR]" to the output of cmd, and still have stderr and stdout separate.

share|improve this answer

With all due credit to @grawity, I'm submitting his comment as an answer, as it seems the best answer here to me.

sed 's/^/[ERROR] /' cmd
share|improve this answer
    
Why is this preferable to the bash solution? –  user14645 Jan 31 '12 at 18:04
    
I suppose it depends on your purpose. If your goal is to simply prepend to every line in a file, this accomplishes that goal with very few characters, using a very familiar tool. I far prefer that to a 10 line bash script. The awk one-liner is nice enough, but I think that more people are familiar with sed than awk. The bash script is good for what it does, but it seems that it is answering a question that was not asked. –  Eric Wilson Jan 31 '12 at 18:22
    
The answer that pjz submitted is also a nice one-liner. It doesn't additional programs, processes and may run a little quicker. –  user14645 Feb 2 '12 at 18:11
    
sed X cmd does read cmd and does not execute it. Either cmd | sed 's/^/[ERROR] /' or sed 's/^/[ERROR] /' <(cmd) or cmd > >(sed 's/^/[ERROR] /'). But beware the latter. Even that this allows you to access the return value of cmd the sed runs in background, so it is likely you see the output after cmd finished. Good for logging into a file, though. And note that awk probably is faster than sed. –  Tino Feb 21 at 23:41

I created a GitHub repository to do some speed tests.

The result is:

  • In the general case, awk is fastest. sed is a bit slower and perl is not much slower than sed. Apparently, all those are highly optimized languages for text processing.
  • In very special situations, where forks dominate, running your script as a compiled ksh script (shcomp) can save even more processing time. In contrast, bash is dead slow compared to compiled ksh scripts.
  • Creating a statically linked binary to beat awk seems not be worth the effort.

In contrast python is dead slow, but I have not tested a compiled case, because it is usually not what you would do in such a scripting case.

Following variants are tested:

while read line; do echo "[TEST] $line"; done
while read -r line; do echo "[TEST] $line"; done
while read -r line; do echo "[TEST]" $line; done
while read -r line; do echo "[TEST]" "$line"; done
sed 's/^/[TEST] /'
awk '{ print "[TEST] " $0 }'
awk -vT="[TEST] " '{ print T $0 }'
awk -vT="[TEST]" '{ print T " " $0 }'
awk -vT="[TEST]" 'BEGIN { T=T " "; } { print T $0 }'
T="[TEST] " awk '{ print ENVIRON["T"] $0 }'
T="[TEST]" awk '{ print ENVIRON["T"] " " $0 }'
T="[TEST]" awk 'BEGIN { T=ENVIRON["T"] " " } { print T $0 }'
perl -ne 'print "[TEST] $_"'

Two binary variants of one of my tools (it is not optimzed for speed, though):

./unbuffered.dynamic -cp'[TEST] ' -q ''
./unbuffered.static -cp'[TEST] ' -q ''

Python buffered:

python -uSc 'import sys
for line in sys.stdin: print "[TEST]",line,'

And Python unbuffered:

python -uSc 'import sys
while 1:
 line = sys.stdin.readline()
 if not line: break
 print "[TEST]",line,'
share|improve this answer
    
awk -v T="[TEST %Y%m%d-%H%M%S] " '{ print strftime(T) $0 }' to output a timestamp –  Tino Feb 22 at 4:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.