I'm trying to centralize logs using fluent. That part seems to be ok, I have a central fluent and other nodes and I use forward plugin to push them. Once on the central fluent I have 2 outputs, one in elasticsearch and one in S3 for archiving purpose. My issue is we do have some applications logs I have to parse before they get in ES and S3. I have tried a couple of configuration but without success. I have no error in td-agent.log, but still the message is not parsed before being pushed in ES. Here is the configuration of my "source" in my fluent.
<source> @type tail path /var/log/xxx/xxx.log tag xxx.scheduled pos_file /tmp/xxx.log.pos format /^\[(?<time>[^\]]*)\] (?<application>[^ ]*)\.(?<level>[^ ]*):(?<log>.*)/ </source>
I verified the regex ... seems to be just fine. Here's a couple of messages I'm trying to parse :
[2018-09-06 19:04:01] CommunicationCron.DEBUG: - Process communication c7891211-9414-4ce7-b44a-cb53d3e2f8a4   [2018-09-06 19:02:01] CommunicationCron.INFO: #### End scheduled cron 2018-09-06 19:02:01  
not really sure what I'm doing wrong here as my regex seems to be working, but i'm not able to get the log properly parsed to be displayed in kibana (through elasticsearch).
Thanks for any hint !