1

I have a mostly default Splunk config that is properly splitting most of my log messages from a standard Java application. We don't override any of the defaults concerning line breaks, line merging, or date formats. In some situations, Splunk appears to be detecting that there is a date on a multi-line log event and incorrectly splits the event on that line. I'd like to know how to prevent this from happening.

An example is included below with 4 log events - each beginning with a date time stamp and severity. Note that the first and 4th events are single-line log events and the second and third are more detailed multi-line log events that include more detail.

In the case of the second event, Splunk correctly splits this event in its entirety. For the third event, however, the date we write next to "occurred on" is off by a millisecond or two from the timestamp at the beginning of the event. When this happens, Splunk splits these events into multiple events.

It's worth noting that the line separator for the multi-line events is likely \n. I'm not 100% sure what the line separator is at the end of each of the events but it's also likely \n.

Is there any way to prevent Splunk from detecting a date/timestamp in the middle of a line and splitting the event based on that token?

2021-08-27 20:57:34,860 ERROR [<redacted>][<redacted>] <redacted>
2021-08-27 20:56:24,118 ERROR [<redacted>][<redacted>] MESSAGE="
<redacted - more info>
<redacted - more info>
--     OCCURRED ON:     08/27/2021 20:56:24:11
<redacted - more info>
<redacted - more info>
<redacted - more info>
<redacted - more info>
<redacted - more info>
2021-08-27 20:56:11,221 ERROR [<redacted>][<redacted>] MESSAGE="
<redacted - more info>
<redacted - more info>
--     OCCURRED ON:     08/27/2021 20:56:11:220
<redacted - more info>
<redacted - more info>
<redacted - more info>
<redacted - more info>
<redacted - more info>
2021-08-27 20:57:09,960 ERROR [<redacted>][<redacted>] <redacted>

1 Answer 1

2

Tell Splunk to break events before a date that comes at the beginning of a line. And it would help to tell Splunk the expected format of timestamps:

LINE_BREAKER = ([\r\n]+)\d\d\d\d-\d\d-\d\d
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:S,%3N

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.