Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I would like to be able to split log files according to a pattern found in them.

For example process all logs and look for /(\w+)\s to match /myresource but exclude /myresource/anythingelse, to redirect everything to /var/log/extractedlog/myresource/access.log. I could easily script this using a bit of grep however, trying to do this in real time could make the problem harder. For example, I would like to call the program twice without generating duplicates.

EDIT

Here is a complete code to get something like this working with syslog-ng /etc/syslog-ng/syslog-ng.conf (credits get to the accepted answer):

# no-parse let syslog load any source
source s_unparsed_source {
    file("/var/log/myservice/access.log"
        flags(no-parse));
};

# Just protect the input and avoid syslog-ng header to be added in the final log
template t_preserve_message {
    template("$MSG\n");
    template_escape(no);
};

# This will filter the message only matching the given expression
filter f_match_pattern1 {
    match("\/pattern1");
};

destination d_target1 {
    file("/var/log/target/pattern1/access.log" template(t_preserve_message));
};

# The actual logging instruction which wraps everything
log {
    source(s_unparsed_source);
    filter(f_math_pattern1);
    destination(d_target1);
};
share|improve this question
2  
rsyslog will do this natively. –  Iain Jul 15 at 8:25
2  
Also syslog-ng. –  Jenny D Jul 15 at 8:41

2 Answers 2

up vote 5 down vote accepted

Both rsyslog and syslog-ng (the two usual programs used in GNU/Linux to manage logs) have means to do this.

With syslog-ng, you can define filters that match a regular expression:

filter myfilter {
  not match("regex" value("\/usr\/sbin\/run-crons"))
  and not match("regex" value("vmware-checker"));
}

And you can also use the pattern database, which allows correlation of events and action triggering.

There's also logstash, which has advanced filtering capabilities. Specifically, it has a grep filter:

filter {
  grep {
    match => { "message" => "hello world" }
  }
}
share|improve this answer

Take a look at Logsurfer.

I use syslog-ng for "normal" logging into files and in addition I pipe everything into logsurfer for classification by program and to spot new/unusual messages.

A snippet from my config so I see all openvpn warnings:

'^.{29} .+ openvpn\[[0-9]+\]: (ERROR|WARN)' - - - 0 echo >>lines.openvpn $0
'^.{29} .+ openvpn\[[0-9]+\]: ' - - - 0 ignore
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.