I'm trying to centralize logs using fluent. That part seems to be ok, I have a central fluent and other nodes and I use forward plugin to push them. Once on the central fluent I have 2 outputs, one in elasticsearch and one in S3 for archiving purpose. My issue is we do have some applications logs I have to parse before they get in ES and S3. I have tried a couple of configuration but without success. I have no error in td-agent.log, but still the message is not parsed before being pushed in ES. Here is the configuration of my "source" in my fluent.

  @type tail
  path /var/log/xxx/xxx.log
  tag xxx.scheduled
  pos_file /tmp/xxx.log.pos
  format /^\[(?<time>[^\]]*)\] (?<application>[^ ]*)\.(?<level>[^ ]*):(?<log>.*)/

I verified the regex ... seems to be just fine. Here's a couple of messages I'm trying to parse :

[2018-09-06 19:04:01] CommunicationCron.DEBUG: - Process communication c7891211-9414-4ce7-b44a-cb53d3e2f8a4 [] []

[2018-09-06 19:02:01] CommunicationCron.INFO: #### End scheduled cron 2018-09-06 19:02:01 [] []

not really sure what I'm doing wrong here as my regex seems to be working, but i'm not able to get the log properly parsed to be displayed in kibana (through elasticsearch).

Thanks for any hint !

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.