2

I would like to be able to split log files according to a pattern found in them.

For example process all logs and look for /(\w+)\s to match /myresource but exclude /myresource/anythingelse, to redirect everything to /var/log/extractedlog/myresource/access.log. I could easily script this using a bit of grep however, trying to do this in real time could make the problem harder. For example, I would like to call the program twice without generating duplicates.

EDIT

Here is a complete code to get something like this working with syslog-ng /etc/syslog-ng/syslog-ng.conf (credits get to the accepted answer):

# no-parse let syslog load any source
source s_unparsed_source {
    file("/var/log/myservice/access.log"
        flags(no-parse));
};

# Just protect the input and avoid syslog-ng header to be added in the final log
template t_preserve_message {
    template("$MSG\n");
    template_escape(no);
};

# This will filter the message only matching the given expression
filter f_match_pattern1 {
    match("\/pattern1");
};

destination d_target1 {
    file("/var/log/target/pattern1/access.log" template(t_preserve_message));
};

# The actual logging instruction which wraps everything
log {
    source(s_unparsed_source);
    filter(f_math_pattern1);
    destination(d_target1);
};
2
  • 2
    rsyslog will do this natively.
    – user9517
    Jul 15, 2014 at 8:25
  • 2
    Also syslog-ng.
    – Jenny D
    Jul 15, 2014 at 8:41

2 Answers 2

5

Both rsyslog and syslog-ng (the two usual programs used in GNU/Linux to manage logs) have means to do this.

With syslog-ng, you can define filters that match a regular expression:

filter myfilter {
  not match("regex" value("\/usr\/sbin\/run-crons"))
  and not match("regex" value("vmware-checker"));
}

And you can also use the pattern database, which allows correlation of events and action triggering.

There's also logstash, which has advanced filtering capabilities. Specifically, it has a grep filter:

filter {
  grep {
    match => { "message" => "hello world" }
  }
}
0

Take a look at Logsurfer.

I use syslog-ng for "normal" logging into files and in addition I pipe everything into logsurfer for classification by program and to spot new/unusual messages.

A snippet from my config so I see all openvpn warnings:

'^.{29} .+ openvpn\[[0-9]+\]: (ERROR|WARN)' - - - 0 echo >>lines.openvpn $0
'^.{29} .+ openvpn\[[0-9]+\]: ' - - - 0 ignore

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .