Using rsyslog to create different log files for different processes

Question

Scenario: I am running a cluster of machines. Each machine runs various python programs with a unique (across the cluster), but dynamically set, ID.

Right now, they are all logging locally. So, I might have logs that look like:

process_5.log
process_6.log

for processes that had ID's 5 and 6.

Another machine may have:

process_20.log
process_25.log

I wish to forward these logs to a logserver running rsyslogd. Python's logging facility has a nice syslog handler, so I understand how I could connect to the remote server. What I haven't figured out is how to use templating/DynFile to maintain log separation.

e.g. on the logserver, I will want to see:

process_5.log
process_6.log
process_20.log
process_25.log

which correspond to the logs of the same name on the sending machine.

Is there a way to pull this off with rsyslogd templating?

Answer 1

Each machine runs various python programs with a unique (across the cluster), but dynamically set, ID.

Does the process_$ID is also the program name? And if so, try this:

$template DynaFile,"/var/log/rsyslog/python_apps/%PROGRAMNAME%.log"
if $programname startswith 'process' then -?DynaFile

You can put it in separate folder with %HOSTNAME% if you want.

PS: I haven't tested. Let me know if it works for you.