We don't have any log management aside from every now and then going in and clearing it out. We have a server that doesn't get too much traffic (most logs are from our live chat). But it made me wonder, what kind of log management procedures should be in place to manage logs properly? Do you set it up so that only specific things are logged, that logs are wiped so many days? How long should access logs be kept? How long should other logs be kept?
Even my logfiles get big because of its. I use the standard logrotate available with almost any linux distro.
I have configured it to logrotate the stuff which gets too big, daily and others weekly, so on.
Give it a try. I even compress them with xz, so the file is relatively small and can be downloaded to your personal box for scrutiny without much loss of bandwidth on either side.
See man logrotate and man logrotate.conf for more info.
First understand that your webserver appends to your access.log so you really should keep it's size to a minimum. My apache servers start to load pages more slowly when they have log files that are over 4 GB (I'm probably the only one that notices).
Give awstats a try. It'll create html based files that will allow you to get great views of your traffic and it'll also clear out that huge access.log file that you have. The first time you run this app it's going to take a really long time since that access.log is so large.
If awstats or logrotate won't do for you, you could always just nuke your access.log once a day with a cron:
echo "" > access.log
But I personally only use this when I have a huge log file that I need to truncate really fast. It's really a waste of logs to truncate them like this.
Use cronolog to create a hourly log file or daily log file. See how to use it.
TransferLog "|/usr/sbin/cronolog /web/logs/%Y/%m/%d/access.log" ErrorLog "|/usr/sbin/cronolog /web/logs/%Y/%m/%d/errors.log"
Using cronjob compress above logs to prevent disk fillup and keep data for few days (may be a month old).
or use logrotate to rotate the logs (As explained in answer above)