My employer has a CustomLog piped handler that reduces and preps the static web content machine access logs into a format that is kosher for being dumped into hadoop/hive. Unfortunately it hinges on a regex to convert the log format along with somewhat convoluted logic to deal with the worst the internet can throw at it.
The fix is to replace the current script and make the entire pipeline more durable. One idea I had was to do away with the regex part and do something like:
LogFormat "%v:%p\t%h\t%l\t%u\t%t\t\"%r\"\t%>s\t%O\t\"%{Referer}i\"\t\"%{User-Agent}i\"" vhost_combined_tabs
The trick is that EVERY field would be tab delimited. So that in a script language, a simple split on "\t" could break the access log down reliably. Even if one of the fields returns as "", the tab would maintain record integrity.
The problem is, I couldn't find anything on google that mentions doing this. To me it seems intuitive to break the work down from the source instead of having a script deal with the problem. Am I missing something here?