I have https_acces_log log files being generated by Apache that a certain web app is logging unwanted data into the file. I can't stop the app from logging the data, so I'd like to write a perl/python script which will take the data output by Apache and run a regex on it to replace the data in real time.

Assuming Perl is the right tool, here's where I'm at so far. Script produces no output right now. Help anyone?

$|=1;                               # Use unbuffered output
while (<STDIN>)                     # Loop through STDIN
    $Msg = $_;                      # Capture the line of input

    if ($Msg =~ m/&passwd=\w+GET/ )         #   Look for the string I don't want
        $Msg =~ s/&passwd=\w+GET/&password=XXXXXXGET/g;
        print $Msg;                 # Print it
        print $Msg\n;

I may be mistaken here, but I believe you would need to have your script handle opening and writing to the log file you are expecting. Apache would not capture the stdout of your script and place it in the log file.

I don't run my own custom scripts with piped logs, but I do use rotatelogs and you do have to give it the path to the log file to write. Such as:

CustomLog "|/usr/sbin/rotatelogs /var/log/httpd/ssl_access_log 86400" common

| improve this answer | |
  • Thanks a ton for the above. I worked out a script that does the trick, though I can't "answer my own question" to post it, as I don't have sufficient brownie points. I'll post it in a few hours when the restriction comes off. Thanks again. – Tad Jun 29 '11 at 17:57
  • Happy to help, could you at least upvote my answer then? – TCampbell Jun 29 '11 at 18:03
  • Unfortunately, I require "15 reputation" to do that, else I'd absolutely do so. – Tad Jun 29 '11 at 18:18

You have an error on the last print statement - you can't use \n without quoting it. It's not needed though, as $Msg should still have a newline, so you can just use print $Msg. If you're using -w this will trigger a warning ("unopened filehandle"), so lines not matching your pattern will not be printed at all.

Removing that your script seems to work - that is, it replaces &passwd=testGET with &password=XXXXXXGET. Is that what you need?

Edit: I took it you were trying this in the command line prior to actually trying to use it on Apache. See @TCampbell's answer if you're testing it in Apache already.

Edit 2: Do use perl -w or use warnings if you're not doing it already. Also consider use strict. You'll be a happier person overall.

| improve this answer | |

Well if you need to do so, there is an example on how to create pipe in syslog:

First create a named pipe using mkfifo:

mkfifo -p /home/data/syslog.pipe

Make syslog.conf to points to this file:

*.info |/home/data/syslog.pipe
Restart syslog:

sudo pkill -HUP syslogd
Create processing script that read the pipe
$ cat > foo
cat /home/data/syslog.pipe | while read input
echo "Do some stuff" ${input}

But be careful, you may have no mangled messages if they arrive faster than our program can process them.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.