Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm using Zabbix's script to push exception stack traces to my remote Zabbix monitoring server. requires a key and a value for any data that it sends. It can read data from stdin, but this overrides any key variable specified. Because my stdin data isn't formatted like Zabbix expects, I need to pass in "value" as an argument. Hopefully this provides some context.

What I want to accomplish is capturing the multiline result from grep into a variable, preserving newlines, so that I can call the script with that variable as an argument.

What I've tried so far looks like this:

tail -Fn0 /var/log/uwsgi.log | grep "Exception:" -A 100 | (read tback; /usr/local/zabbix/bin/zabbix_sender -z -s MyHostName -k uwsgi_traceback -o $tback)

As best I can tell, that never invokes

To test, I've tried using this command, which doesn't seem to work either:

tail -Fn0 /var/log/uwsgi.log | grep "Exception:" -A 100 | (read errorlines; echo "$errorlines" > /tmp/errorlines.txt)

The /tmp/errorlines.txt file is never created.

How can I capture grep's output lines into a variable so that I can call another script with that variable as an argument?

share|improve this question
Why don't you use log file monitoring? – Michael Hampton Apr 5 '14 at 17:09
Zabbix's built in log file monitoring is great at alerting me that something went wrong, but I haven't managed to get it to record any surrounding lines. Even using the output argument to log, I can't seem t make it capture more than the line that triggered it. If you know of a way to make this work, please share it with me. – skyler Apr 5 '14 at 17:53
up vote 1 down vote accepted

The problem is the tail part; since it is in continuos mode it wont ever spit out something for "read" to read.

This should work:

echo "0" >/tmp/numberoflines
while [ 1 ]
    NUMBER=$(cat /tmp/numberoflines)
    LINES=$(wc -l < /var/log/uwsgi.log)

    if [ $DIFFERENCE != 0 ]; then
            exception=$(tail -n $DIFFERENCE /var/log/uwsgi.log | grep "Exception:" -A 100)
           /zabbix/bin/zabbix_sender -z -s MyHostName -k uwsgi_traceback -o $tback) $exception;

sleep 5;
echo "$LINES" >/tmp/numberoflines
share|improve this answer

Here my approach.

  1. Use zabbix "log" type key to monitor error pattern in a log-file (/var/log/uwsgi.log is the case here).
  2. invoke zabbix remote command triggered by the above 1. This remote command fetch lines surrounding the error by linux command tail(1).

The advantages of this approach are:

  1. No need to install&setup agent-host side special script (like discussed above). zabbix 'log' type item has been already provided such a kind of purpose.
  2. no script on agent-host mean that there is no additional CPU & memory consumption. Fetching several lines surounding error in the log happens only at trigger.

Let me explain the detail how to setup below:

  1. register regular expression in zabbix from Administration > General > Regular expressions > [New regular expression] like "error|fail|fatal". Let's assume the variable name is @uwsgi_error_pattern.
  2. register item from Configuration > Hosts > [Target host] line > Items > [Create Item] with the following attributes:
    • Description: [any name]
    • Type: Zabbix agent(active)
    • Key: log[/var/log/uwsgi.log,@uwsgi_error_pattern]
    • Type of information: Log
  3. register 2nd item from Configuration > Hosts > [Target host] line > Items > [Create Item] with the following attributes, to accept fetched log flagment sent by zabbix remote command(mentioned later):
    • Description: [any name]
    • Type: Zabbix trapper
    • Key: my_app.fetch_uwsgi_log
    • Type of information: Text
  4. register trigger from Configuration > Hosts > [Target host] line > Triggers > [Create trigger] with the following attributes:
    • Name: uwsgi log monitor on {HOSTNAME}
    • Expression ({[Target host]:log[/var/log/uwsgi.log,@uwsgi_error_pattern].iregexp(@uwsgi_error_pattern)})#0&({[Target host]:log[/var/log/uwsgi.log,@uwsgi_error_pattern].nodata(300)})=0
  5. register action which remotely executes to fetch data surrounding the error line of the log file as follows:
    • Name: Fetch latest uwsgi log on error
    • Action Operations:
      • Operation Type: Remote command
      • Remote command: {HOSTNAME}:zabbix_sender -z [zabbix-server] -s {HOSTNAME} -k my_app.fetch_uwsgi_log -o "`tail -200 /var/log/uwsgi.log`"

NOTE-1: key name "my_app.fetch_uwsgi_log" in the above step is just example. We can define any unique name to bind zabbix_sender and the item.

NOTE-2: You may need AllowRoot=1 on /etc/zabbix/zabbix_agentd.conf to allow zabbix-agent read uwsgi.log.

share|improve this answer

You must turn off word splitting in bash, eg, by clearing IFS:

export IFS=""
set NEWVAR=`your tail|grep expression`

Now echo $NEWVAR has the newlines.

share|improve this answer
This will run once and capture tail's output, no? Will it follow the file as it's written to by the app that I'm logging? – skyler Apr 5 '14 at 17:57
Unfortunately that didn't work export IFS="" tail -Fn0 /var/log/uwsgi.log | grep "Exception:" -A 100 | (read errorlines; echo "$errorlines" > /tmp/errorlines.txt) – skyler Apr 5 '14 at 18:02
I marked up this answer because I didn't know about the IFS trick for capturing multiline output into a variable. – pdwalker Jan 14 '15 at 1:42

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.