I would like to be able to split log files according to a pattern found in them.
For example process all logs and look for /(\w+)\s
to match /myresource
but exclude /myresource/anythingelse
, to redirect everything to /var/log/extractedlog/myresource/access.log
. I could easily script this using a bit of grep however, trying to do this in real time could make the problem harder. For example, I would like to call the program twice without generating duplicates.
EDIT
Here is a complete code to get something like this working with syslog-ng /etc/syslog-ng/syslog-ng.conf
(credits get to the accepted answer):
# no-parse let syslog load any source
source s_unparsed_source {
file("/var/log/myservice/access.log"
flags(no-parse));
};
# Just protect the input and avoid syslog-ng header to be added in the final log
template t_preserve_message {
template("$MSG\n");
template_escape(no);
};
# This will filter the message only matching the given expression
filter f_match_pattern1 {
match("\/pattern1");
};
destination d_target1 {
file("/var/log/target/pattern1/access.log" template(t_preserve_message));
};
# The actual logging instruction which wraps everything
log {
source(s_unparsed_source);
filter(f_math_pattern1);
destination(d_target1);
};