How do i remove ip addresses from log files

Question

I have an apache / nginx whatever webserver which logs client ip addresses to the access logs. Now these log files are rotated via logrotate.

I want to keep the IP addresses for some days, then - after 7 days, i want to remove the ips from the log files for privacy reasons (mostly dictated by german law)

using mod_removeip or something like that doesn't work because I need to filter some requests based on their IP addresses.

Is there any 'standard' way how to do that - maybe even with logrotate?

EDIT

i just found this script http://www.zendas.de/technik/sicherheit/apache/index.html but it depends on the ability to pipe all logging trough the script in real-time. i'm not really sure about the performance implication of this approach.

Also, this only works for the 'front-end' server logs, not the application server logs

Answer 1

PCRE! (Perl-Compatible Regular Expression)

s/\b(1?[0-9]{1,2}|2[0-4][0-9]|25[0-5])\.(1?[0-9]{1,2}|2[0-4][0-9]|25[0-5])\.(1?[0-9]{1,2}|2[0-4][0-9]|25[0-5])\.(1?[0-9]{1,2}|2[0-4][0-9]|25[0-5])\b/REMOVED IP/g

Use that as a filter in a perl script or any other suitable language (quite a few use PCRE or some other close-enough regex language that will work) to rewrite your log files at 7 days.

$ cat > file_with_ip
some text from 192.168.1.1
^D
$ perl -p -i -e 's/\b(1?[0-9]{1,2}|2[0-4][0-9]|25[0-5])\.(1?[0-9]{1,2}|2[0-4][0-9]|25[0-5])\.(1?[0-9]{1,2}|2[0-4][0-9]|25[0-5])\.(1?[0-9]{1,2}|2[0-4][0-9]|25[0-5])\b/REMOVED IP/g' file_with_ip
$ cat file_with_ip
some text from REMOVED IP

Answer 2

I don't think logrotate will do it; you may need to look at creating a script that will decompress the files, process them through awk or sed to strip the IP's out, then recompress them. Just can't do it on "active" log files.