Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I just changed my Apache log behavior because it was making me having very BIG files...

So I now use cronolog to split my logs to log/httpd/2012/11/access_2012.11.30.log for exemple, pattern : %Y/%m/access_%Y.%m.%d.log

I now want to split my old 42GB file to the same structure but really don't know how to do that efficiently.

I tried some simple commands with cat, egrep, awk... but really don't know how to handle all that in a more powerful script.

Here is how the log looks like :

x.x.237.134 - - [08/Apr/2011:14:43:09 +0200] "GET...
x.x.50.15 - - [08/Apr/2011:14:43:09 +0200] "GET...
[...]
x.x.254.19 - - [28/Feb/2012:15:24:48 +0100] "GET...

So I need for yeah line to get :

  • year %Y (ex. 2012)
  • month %m (ex. 11)
  • day %d

And to push out the entire line to : %Y/%m/access_%Y.%m.%d.log

Can someone give me clues to get that working ?

Thanks a lot for your interest.

share|improve this question

1 Answer 1

You could grab one of the existing Apache access log splitters from the interweb, alternatively you could hack something together in a few lines of sh, php, phyton, or perl e.g.

#!/usr/bin/perl
# Purpose: Split an Apache HTTPD access log up into daily files, with each file placed
#          in a year and month specific directory.
#
my @aFields;
my $sLogDir='/var/log/httpd';
my $sCurrentDay="1901.01.01";
my $sCurrentMonth="1901/01";
my %mon2num = qw(
  Jan 01  Feb 02  Mar 03  Apr 04  May 05  Jun 06
  Jul 07  Aug 08  Sep 09  Oct 10  Nov 11  Dec 12
);
open (FILE,"$sLogDir/access_log") || die("Error: Could not open file - $sLogDir/access_log !");
while (my $sLine = <FILE>) {
  @aFields=split(' ', $sLine);
  @aFields=split(':', $aFields[3]);
  @aFields=split('/', $aFields[0]);
  $aFields[0] =~ s/^\[//;
  $aFields[1] = $mon2num{$aFields[1]};
  if ($sCurrentMonth !~ /$aFields[2]\/$aFields[1]/) {
    print "Processing: $aFields[2]/$aFields[1]\n";
    $sCurrentMonth = "$aFields[2]/$aFields[1]";
    unless(-d "$sLogDir/$aFields[2]"){
      mkdir "$sLogDir/$aFields[2]" or die("Error: Couldn't create - $sLogDir/$aFields[2]");
    }
    unless(-d "$sLogDir/$sCurrentMonth"){
      mkdir "$sLogDir/$sCurrentMonth" or die("Error: Couldn't create - $sLogDir/$sCurrentMonth");
    }
  }
  if ($sCurrentDay !=  "$aFields[2].$aFields[1].$aFields[0]") {
    $sCurrentDay = "$aFields[2].$aFields[1].$aFields[0]";
    close(NEW);
    open (NEW,">>$sLogDir/$sCurrentMonth/access_${sCurrentDay}.log") || die("Could not open file - $sLogDir/$sCurrentMonth/access_${sCurrentDay}.log !");
    print NEW $sLine;
  }
}
close(FILE);
close(NEW);
print "All done :o)\n\n";
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.