2

My employer has a CustomLog piped handler that reduces and preps the static web content machine access logs into a format that is kosher for being dumped into hadoop/hive. Unfortunately it hinges on a regex to convert the log format along with somewhat convoluted logic to deal with the worst the internet can throw at it.

The fix is to replace the current script and make the entire pipeline more durable. One idea I had was to do away with the regex part and do something like: 

LogFormat "%v:%p\t%h\t%l\t%u\t%t\t\"%r\"\t%>s\t%O\t\"%{Referer}i\"\t\"%{User-Agent}i\"" vhost_combined_tabs

The trick is that EVERY field would be tab delimited. So that in a script language, a simple split on "\t" could break the access log down reliably. Even if one of the fields returns as "", the tab would maintain record integrity.

The problem is, I couldn't find anything on google that mentions doing this. To me it seems intuitive to break the work down from the source instead of having a script deal with the problem. Am I missing something here?

Improve this question
2
  • The last sentence of your first para ends mid-sentence: "In the last two weeks I".
    – Mark Wagner
    Oct 19, 2010 at 20:52
  • @embobo Thanks, was going to elaborate that the current implementation has hung my companies entire static web array in the last few weeks so I have to replace it and want to improve on it.
    – David
    Oct 19, 2010 at 20:57

1 Answer 1

Reset to default
2

I don't think you are missing anything, the obvious issue is that none of the unquoted data fields must have any possibility of containing the separator character. If any quoted data fields might ever contain your separator character then you'll have to use a less efficient method to subsequently separate the fields.

Tab ought to be relatively safe

Note that in httpd 2.0 versions prior to 2.0.46, no escaping was performed on the strings from %...r, %...i and %...o. This was mainly to comply with the requirements of the Common Log Format. This implied that clients could insert control characters into the log, so you had to be quite careful when dealing with raw log files.

For security reasons, starting with 2.0.46, non-printable and other special characters are escaped mostly by using \xhh sequences, where hh stands for the hexadecimal representation of the raw byte. Exceptions from this rule are " and \ which are escaped by prepending a backslash, and all whitespace characters which are written in their C-style notation (\n, \t etc).

http://httpd.apache.org/docs/2.0/mod/mod_log_config.html#formats

Improve this answer
1
  • Thanks! I figure for the edge cases where a query string does contain a tab character somehow, I'd throw the entire line away ( hoping this only happens 1x in say a million records ).
    – David
    Oct 21, 2010 at 0:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .