I'm trying to centralise logging in an environment that using multiple application technologies (Java, Rails and various DBs).
We want to developers to bring up stacks with Docker Compose, but we want to them to refer to a central log source (ELK) to debug issues, rather than trying to open shells into running Docker containers.
The applications all write to the file system rather than to STDOUT/STDERR, which removes all of the options associated with the Docker logging driver, and logspout too.
What we have done is configure the containers to have rsyslog include the application log files and forward those to logstash which has a syslog input. This works in terms of moving the logs from A to B, but managing multi-technology logs in ELK based on the syslog input is horrible (eg trying to capture multine Java stacktraces, or MySQL slow queries).
Is there a better way to do this? Should I be running logstash in each container, so that I can apply filters and codecs directly to the log files, so that I don't have to rely on the syslog input?
Of is there some way to use the Docker logging driver with application log files that are written to the file system?