Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We currently use logger to get our Apache access logs to syslog-ng with a line like this one in each vhost:

CustomLog "|/usr/bin/logger -p local1.info  -t www_main" combined

It seems that almost every tutorial or advice I can find regarding Apache and syslog-ng uses this method to get the logs into syslog-ng. (Example). The rest use named pipes and the pipe() source. (Example).

The trouble with this is that logger splits any lines longer than 1024 bytes and sends them as separate log entries, meaning that some log entries end up on separate lines in the final destination log file on the logging box.

Using a named pipe and the pipe() source in syslog-ng solves the split lines problem but comes with its own set of minor problems and annoyances. To name a few, the named pipe has to be created before both Apache and syslog-ng are started, syslog-ng must be started before Apache is started and tagging the logs (which is done above with -t) must now be done in the syslog-ng configuration file rather than in the vhost.

This page suggests writing a short Perl script to use in place of /usr/bin/logger.

I was wondering if anyone knows of any alternatives to logger, preferably written in a natively compiled language or maybe an updated version of logger that doesn't split long lines in half.

I would also be interested in hearing of any other solutions to the problem of long log lines being split, such as a way of combining them back together using syslog-ng once they reach the logs servers.

share|improve this question
add comment

2 Answers 2

up vote 3 down vote accepted

Updating my answer after looking into this more.

This seems to a limit with /usr/bin/logger, which is expected to conform with the syslog RFCs. http://www.faqs.org/rfcs/rfc3164.html says:

The total length of the packet MUST be 1024 bytes or less.

If you to send more then 1024 characters to syslog via the commandline (outside of Apache), you'll run into this same limit.

Keep in mind that the 1024 character limit probably exists elsewhere. I think the maximum size for a HTTP GET is 1024 characters, and I seem to recall that some printf library routines have a hard limit of 1024 characters (There was a security alert a couple years ago regarding the 1024 character limit regarding some syslog/ string printing utilities, if I remember right). So, it seems that your options are:

3) Try to stop your HTTP applications from writting long log messages. This is easier said then done. 1) Recompile logger and increase this limit. If you do this, keep in mind that you're changing a core utility and this may result in unexpected behavior. To mitigate this, put this utility in /usr/local/bin or /opt/bin. Do not replace /usr/bin/logger. 2) Don't send from Apache to syslog. Something like the following should work around the 1024-character limit, since this doesn't use syslog.

CustomLog logs/access_log

4) http://www.oreillynet.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html uses sys::syslog and seems to be a reasonable alternative to /usr/bin/logger. You need to check sys::syslog for this same 1024 character limit. It's Perl, and should be easy to override.

Old answer:

It looks like this limit is tunable within syslog-ng, according to http://www.campin.net/syslog-ng/faq.html

syslog defaults to 1024 byte long messages, but this value is tunable in syslog-ng 1.5 where you can set it to a higher value.

options { log_msg_size(8192); };

share|improve this answer
    
I didn't mention this in the question but I did already find the syslog-ng log_msg_size() option. Unfortunately the messages are being split by logger before they get to syslog-ng so that option makes no difference. That RFC was very interesting, particularly the bits about truncating anything over 1024 bytes. It seems that I am looking for something that ignores that part of the RFC. I noticed the author of syslog-ng in the list of credits at the bottom. It's interesting that when I use a named pipe instead of /usr/bin/logger, those bits of the RFC are ignored. –  Ladadadada Nov 10 '11 at 18:08
    
Ah, I see that the syslog-ng 2.0 manual, log_msg_size seems to default to 8192. –  Stefan Lasiewski Nov 10 '11 at 18:27
    
@Ladadadada : I updated my answer with more information. –  Stefan Lasiewski Nov 11 '11 at 18:23
1  
A quick C implementation of the Perl script described in the oreillynet article can be found at wiki.rsyslog.com/index.php/… It seems to be much faster. –  Tonin Sep 28 '12 at 15:01
add comment

I personally go for InterSect Alliance's free products for this. They have a dedicated Apache product - albeit old - and a more general product that can slurp any text based logs to syslog format.

http://www.intersectalliance.com/projects/EpilogUNIX/index.html

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.