What you seem to wanting is a component called a 'log shipper' (not to be confused with database-replication). Logstash itself supplies one; it is referred to as the 'agent', and it is written in Java. If you don't want to install [a particular version of] Java on all of your servers (which is a quite-reasonable objection), then there are plenty of alternatives.
Log-shippers have a few common properties which you can compare them on:
- they generally can tail files in (near) real-time, either polling (common) or using inotify;
- they will generally cope well with log-rotation events;
- they will often be implemented in one of many languages;
- they will be able to read from (many) various sources, including various types of databases, or the windows event log;
- commercial support options are often available, as well as free options; the commercial versions of the free ones may offer more input options;
- they differ on cross-platform availability;
- licence costs for different platforms may vary wildly (eg. rsyslog on Windows versus Linux)
- scalability and footprint made vary significantly;
- they may (not) be able to transmit logs over SSL;
- they may (not) be able to buffer events if the receiver is unavailable;
- they may (not) be able to transform/filter messages;
- they have different levels of scriptability;
The Logstash Book lists a few; such as lumberjack (written in Go, and not packaged by my OS), and there are plenty others.
rsyslog can also read from files, but the licensing costs for Windows will likely put you off; but this can be useful if you have a RHEL/Centos/other appliance that happens to have rsyslog installed (being an appliance you generally don't have the freedom of installing whatever you like, for supportability reasons).
I'm currently using nxlog, which is written in C; its reasonably small, doesn't have annoying runtime or build dependencies; runs on Linux and Windows at least; fairly well ported; easy to build on platforms that don't have a package provided; has fairy good documentation (but could still be better); a responsive developer; commercial support available; can transmit over SSL; reads from a bunch of different inputs, including files, syslog, windows event log, and others; does some processing, including applying a bunch of tags for later processing in logstash; outputs to a whole bunch of different things (I'm sending mine to a central log-server, which buffers, writes to disk, rotates the files, and also sends on the data to Logstash for ingestion into Elastic Search)
Nxlog isn't perfect; it has a number of bugs, particularly around the configuration parsing (watch out for missing semicolons), and the semantics of the configuration can get a bit confusing when you want to try and do something more complex. I've bumped into at least one memory leak, and one other issue which requires me to implement some hung-process detection for the meantime, but all that aside, I'm still reasonably happy with our use of nxlog so far for our fleet thus far, and we're pushing a lot of data through it.
However, since you did explicitly mention using SSH (is that a real requirement?), you could set up a authorized-keys scp of a file, and then have logstash feed on it locally; and (assuming you call logstash afresh on it each time) make use of logstash's ability to record where in the file it got up to. One thing to be careful about though, you probably don't want the inode to change, so overwrite the file by using '... > file.txt', which will replace the contents, but not replace the file itself. However, that will potentially get frustrated by log-rotation. You really want a log-shipper on the remote machine. Using rsyslog can be useful if there is a requirement for minimal-touch of an environment, although you won't have the ability to transmit over SSL (although perhaps an updated version of rsyslog will allow that).
Hope that helps,
Cameron
