rsync destination based on file date

Question

I have a large directory of log files on a server. All the logs are in one big directory.

We would like to back these up to another host (our archive server), and only keep the last 90 days of backups on the 'live' server.

however, due to the sheer number of logs that we generate with this application, we would create a massive folder that would be hard to browse. Is it possible, with RSYNC to specify the destination folders based on the date of the file (not today's current date)..

For example..

rsync <options> /logs/*.log ArchiveServer:/archive/ServerA/logs/<YYYY>/<MM>
where application.20111230.log gets dumped to /archive/ServerA/logs/2011/12
and application.20120301.log gets dumped to /archive/ServerA/logs/2012/03

I could dump them all into a generic folder, like /archive/ServerA/Current/ and then, maybe weekly look for files older than 90 days and sort them then.. but doing it directly in rsync seems like it would be much cleaner (and I wouldn't have two places to look for logs depending on how new it is)

Answer 1

Something like this?

rsync /logs/*.$(date '+%Y%m%d').log ArchiveServer:/archive/ServerA/logs/$(date '+%Y')/$(date '+%m')/

Update:

Right, so the date is already in the log filename. I should have read more slowly.

I don't think you would do this directly with rsync, but would wrap things up in a script. If you want to bang things through in a first run, something like:

for daysago in `seq 0 90`
do
  rsync /logs/*.$(date '+%Y%m%d' -d "$daysago days ago").log ArchiveServer:/archive/ServerA/logs/$(date '+%Y' -d "$daysago days ago")/$(date '+%m' -d "$daysago days ago")/
done

You can stick an "echo" in front of the rsync command in the loop to see if that's what you want.

Whatever you do will incorporate those elements, though other people will be more clever and elegant about the shell scripting than me.