A log collector using sftp and log rotation using any language

Question

I have to collect logs from remote solaris machines on almost real time basis.

The collection needs to be agent less meaning I cannot change any configuration on the machines generating logs.

I am thinking of using sftp to download the files on periodic basis but I don't want to download the whole file, I want to just download the part that has changed.

So I thought I will take the size of file. Download the file. Next time, I will start downloading from the position i downloaded last time. But now the issue is how to tackle file rotation. I will note down the last modified time, before downloading, and if the size of the current file is less than position i downloaded from last time, i will download from the .0 file.

Another issue was, I needed to upload the log to a database server. So I need a way that I only download complete enteries.

If anybody has any suggestions please do tell me or if you know of a program I should use, please point it out to me.

Answer 1

try to use sshfs. maybe it helps.

Answer 2

Set up Keys (with no passphrase, so you can put it in a script), then run rsync over an SSH link. A script can do this for you, running from cron. See http://troy.jdmz.net/rsync/index.html

rsync will transfer over just the changed blocks in a file. When it reports that no more blocks were transferred, then you know you've got the whole file -- so it's ready to send up to your database server.