Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Using rsyslog v8.2.2, I would like to group multi-line messages, such as those output by the MySQL Slow Query log.

Each new message begins with three consecutive lines having '#' as the first char of the line, though some messages have only two lines having a '#' at the beginning of the message. (I suspect queries triggered by PHP cli, instead of standard PHP, do not carry the # Time: stamp). Example:

# Time: 140817  0:59:22
# User@Host: root[root] @ localhost []
# Query_time: 5.864315  Lock_time: 0.000033 Rows_sent: 857715  Rows_examined: 857715
SET timestamp=1408237162;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `sales_flat_quote_shipping_rate`;
# Time: 140817  1:00:06
# User@Host: user2[user2] @ localhost []
# Query_time: 4.070595  Lock_time: 0.000068 Rows_sent: 0  Rows_examined: 1536
use db2;
SET timestamp=1408237206;
DELETE FROM `catalog_product_index_price_idx`;
# User@Host: db3[db3] @ localhost []
# Query_time: 3.892674  Lock_time: 0.046493 Rows_sent: 0  Rows_examined: 659
use db3;
SET timestamp=1408237206;
UPDATE user SET cert = 'yes' WHERE site_id < 10;
# Time: 140817  1:00:06
# User@Host: user2[user2] @ localhost []
# Query_time: 4.070595  Lock_time: 0.000068 Rows_sent: 0  Rows_examined: 1536
use db2;
SET timestamp=1408237206;
DELETE FROM `catalog_product_index_price_idx`;
# User@Host: db3[db3] @ localhost []
# Query_time: 3.892674  Lock_time: 0.046493 Rows_sent: 0  Rows_examined: 659
use db3;
SET timestamp=1408237206;
UPDATE user SET cert = 'yes' WHERE site_id < 10;

How do I configure rsyslog to send them over as a single message?

share|improve this question
    
While there are ways around this, SYSLOG is NOT designed for this. I am not sure if it is even possible in this specific case. –  Florin Asavoaie Aug 17 '14 at 19:31
    
I will take SYSLOG out of the tags. Thanks. –  Gaia Aug 17 '14 at 20:13

1 Answer 1

up vote 5 down vote accepted

Rsyslog can group multiline log message into a single message via imfile module, however only following three read modes are supported:

  • Line based (default) - each line is a new message.
  • Paragraph - there is a blank line between log messages.
  • Indented - new log messages start at the beginning of a line. If a line starts with a space it is part of the log message before it.

Your case doesn't match any of these, so you can't do the grouping directly via rsyslog. However you can parse your log files and convert them to a format which rsyslog will be able to understand. Simple shell script may do the job or you can have a look at fully featured log management tools like logstash.

share|improve this answer
    
any ideas on how such shell script would work? –  Gaia Aug 17 '14 at 20:19
1  
This is worth a separate question, but a quick and simple algorithm would be to insert an empty line above the first # of each group of lines starting with #, starting with a second group. –  grekasius Aug 17 '14 at 20:23
    
In other words, the script has to insert a blank line before every line that begins with '#' that is followed by another line beginning with "#". –  Gaia Aug 18 '14 at 15:59
    
Not really, because in case of 3 consecutive lines starting with #, based on proposed logic, you will insert two blank lines. –  grekasius Aug 18 '14 at 16:07
1  
Easier way to think about this might be, if you have a line which doesn't start with #, but is followed by a line which starts with #, insert a blank line between them. –  grekasius Aug 18 '14 at 16:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.