Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have several applications which are generating their own plain-text log files, which I would like to forward to a remote syslog server for centralized logging. I do not have access to root on these machines, nor can I reconfigure syslog to redirect output to a remote machine.

I've found some solutions online, but they're mostly people's homemade bash scripts, and I'm looking for something more robust that is suitable for implementation in a potentially high-volume production environment.

Preferably something designed with an eye for a small footprint, background daemon that keeps running, that can keep up with a lot of lines, etc. - What solutions are currently available?

share|improve this question
2  
Have you looked at the text file input module for rsyslog? –  yoonix Oct 22 '13 at 22:31
    
@yoonix: No I haven't, but I'm going to :) –  Michael Martinez Oct 22 '13 at 22:34
2  
Uhm, syslog can send to remote syslog servers. Configure your local syslog to send to a remote server. Then access your local syslog via the standard syslog calls or by using logger or something. –  Zoredache Oct 22 '13 at 22:41
3  
Why don't you write your log files to a named pipe and have a daemon listening that sends them on their way serverfault.com/questions/189477/… –  Iain Oct 23 '13 at 20:37
2  
You shouldn't have to modify the app just put a named pipe of the same name as the log file the app is writing to in place . –  Iain Oct 23 '13 at 20:51

5 Answers 5

There are a number of ways to tackle this. But the very, very first thing you should do is: forward the logs using syslog itself.

Syslog (and many replacements for syslog) have built-in facilities to forward logging to another syslog server at a different address. You can easily do so by changing the configuration file and appending the address to forward the facility to. For instance, adding this line to:

*.*    @192.168.1.1

...would forward all facilities to the machine at 192.168.1.1, which (hopefully) has the service running. The example I give here is for rsyslog, which is the stock syslog server on Debian, although it should work for many others. Consult the documentation for your implementation of syslog with man syslog and see what it says about "forwarding".

The remote syslog server can be anything you like. There are even products, like Splunk, which happily aggregate these logs into a single view with a web dashboard, search, event-driven notifications, etc. etc. You can see more here: http://www.splunk.com/ If that doesn't meet your needs, you can use something else. There are even syslog servers that will dump to a SQL database!

Sure, you could write your own script/program/service to do this for you, but why re-invent the wheel when it's both done for you and already given to you?


Edit: So I went back and re-read the question, and noticed several comments. It sounds like:

  1. you want to aggregate your application logs
  2. you don't have access to root
  3. your application(s) just dump text somewhere
  4. your application(s) doesn't know how to write to the local syslog
  5. you don't have control over your application source code

So let's address each one in sequence:

  1. syslog was meant to aggregate logs together. You can use anything you like, but there is a reason why it's been around for a long time. It's well-tested, well-debugged, well-documented, well-known, and for most *nix platforms nearly universally supported in one flavor or another.
  2. we don't need access to root to set up logging. We only need access to the syslog API. root is not a requirement to write to the syslog; if this were the case, then all of those services that drop privileges would be unable to write diagnostics to the log files.
  3. Re: text dumps, this is normal. however, you should be able to use a subshell to pipe the output of STDERR and STDOUT to a program that calls the syslog API. This isn't rocket science, it's far from being brittle, and it's well documented. In fact, it's one of the reasons that output redirection even exists. A simple command that could be tossed into a single shell script would be:

    ( my-application 2>&1 | my-syslog-shunt ) &

  4. if you have the ability to alter your application's source code, you should write a shunt into it to dump the text output to syslog instead of a plain text file. This shouldn't be too hard; all you do is take the lines you would output, and wrap them with a call. However....

  5. you might not have access to the source code at all, so you can't do this. Which means something like #3 above would work fine.

share|improve this answer
    
two reasons: (1) simply because, as already mentioned, didn't have root or sudo on the boxes in question. (2) "logger" itself can forward to remote server, but has a 400-character limit per log line, which is not appropriate for Apache logs. Anyway, I already put together a custom solution which does exactly what I needed (and improves "logger" too). See my answer here for "filelogger" –  Michael Martinez Mar 12 at 18:47

I hacked together tail.c and logger.c into a single, small footprint compiled program (binary) that is lightweight, fast and stable. As long as it has read access to the log file(s), then it works without needing root privilege.

I also made a couple improvements to the native logger and added a new (optional) capability of inserting a text string at the beginning of each log line before it gets sent to the log server. The result is a program that can be run by itself, without needing to use shell pipes (i.e. don't need to tail logfile | logger). It will run forever until explicitly killed or it encounters an error writing to the network socket. It even continues to run if the log file is rotated or even disappears (it will just continue to look to see if the file reappears.)

It's easy to use: just give it one or more log file(s) to monitor, and each time a new line gets written to the file, it will send a copy of that line to the local or remote syslog server you specify. Plus the extra text string if you use that option.

I actually finished the program back in December, but was waiting for Yahoo to take copyright and make it available, which they've now done. (I wrote it as part of my job at Yahoo).

filelogger program information and download link:

share|improve this answer
    
@slm: I rewrote as you requested –  Michael Martinez Mar 12 at 18:08

You could use logstash with the file input and syslog output.

For example, create a configuration with the file (or files) you want to monitor and your syslog server info.

file-to-syslog.conf:

input { file { path => "/var/log/kern.log" } }
output {
    syslog {
        facility => "kernel"
        host => "syslog.example.com"
        port => 514
        severity => "informational"
    }
}

The start up logstash with

java -jar logstash-1.2.2-flatjar.jar agent -f file-to-syslog.conf
share|improve this answer
    
+1. if using rsyslog's file input isnt an option, logstash is the next best thing. In many ways it's better in the long run. –  Sirex Oct 29 '13 at 19:12
    
I'm not familiar with this. If it does what I need, it would have saved me the trouble of hacking coreutils and util-linux. –  Michael Martinez Oct 29 '13 at 19:12
    
yeah, the config is going to look a bit like this: pastebin.com/xeC9hxD3 –  Sirex Oct 29 '13 at 19:16
    
looks like a very cool tool, but definitely overkill for what I need here. logstash is its own service, with web interface, requires java, etc. I'll continue to use my filelogger which is lightweight, small footprint, optimized for performance. ... But, thanks for suggesting logstash because I can see a need for it in other situations in the future! –  Michael Martinez Oct 29 '13 at 19:18
    
yeah its a jar packed jruby tool. The gui is actually kibana which is packaged into it for easy but is actually a seperate project, so it's not needed just for parsing messages. It's basically a swiss army knife of logging. You define inputs and outputs and in the middle you can optionally grok the logs, which gives them context. -- IT's likely overkill for you unless you also want to use elasticsearch on your log data. –  Sirex Oct 29 '13 at 19:23

I'm answering my own question.

swatch might have worked, but I was unable to get perl's Sys::Syslog module to work on the host, and the /usr/bin/logger that installed on the host does not support logging to remote server (util-linux-ng-2.17.2).

So, the first thing I did was to download the source code for util-linux-2.20.1 for which the logger program does support remote logging. Upon testing, it became apparent there is a limit imposed on the number of characters allowed on the log line. Digging into the source code I found a hard coded 400-character limit. (If you don't believe me, run "strings /usr/bin/logger | grep 400" on any Linux system).

This limit is not acceptable for apache-type of logging (including nodejs), so I modified the code and increased the limit to 4096. While I was at it, I also added a new command-line option which allows one to insert an optional text string at the beginning of each log line. I did this because the nodejs logs do not include the hostname as one would might see in apache.

At this point, I could run a shell script with "tail -F -n 0 [logfile] | ./modified_logger ...." and it worked. But I had some concerns about running this from supervise (daemontools) or even in the background, because if one or the other sides of the pipe terminates, then there is the risk the whole pipe will terminate. I also had concerns (albeit untested) about performance.

so I decided to combine the tail functionality with the logger functionality into a single executable binary that would bypass the need to use Unix pipes or external programs. I did this by hacking tail.c from gnu coreutils and incorporating what I need into the modified logger program.

The result is a new binary (117k size) which I'm calling "filelogger" and which continously monitors one or more files and logs each new line to a local or remote syslog, either via UDP or TCP. It works like a charm. I was able to do a little benchmarking and it logs about 17,000 lines (1.8MB) in about 3 seconds across subnets with a vlan and a couple physical switches between them, to a remote server running syslog-ng.

to run the program you do something like the following (either in the foreground, background, or supervised with daemontools):

./filelogger -t 'access' -d -p local1.info -n [remote loghost] -u /tmp/ignored -a $(hostname) /tmp/myfile1 /tmp/myfile2 ...

/tmp/myfile1 and /tmp/myfile2 are the files being monitored.

The "-a" is the new option I added. In this case I insert the local hostname at the beginning of each log line.

This solutions was exactly the type of solution I was looking for when I asked the question and, as it turned out, did not exist until I made it myself. :)

share|improve this answer
    
I'll probably make this available on sourceforge at some point. It's advantages are that its a very small footprint, lightweight, easy to use, and optimized for performance. Once message text is read, all processing is done in memory buffer then transferred directly to socket. –  Michael Martinez Oct 29 '13 at 19:21
1  
xkcd.com/763 –  Iain Oct 30 '13 at 19:35
4  
I'm trying not to be harsh, but I am going to be blunt: This solution didn't exist because it's terrible. Rather than interfacing with the other groups in your organization and implementing a sane, standard solution you've thrown a hack in place with totally unsupported code that you now need to test/debug/maintain going forward. You ignored easily 50+ years of combined experience telling you "Don't do that" -- I hope for your sake this doesn't blow up in your face, but you are absolutely, unquestionably Doing It Wrong here... –  voretaq7 Oct 30 '13 at 19:55
    
yeah. right.... This is how open source moves forward, dude. If everyone did it your way, there'd be no progress. How do you think GNU, Linux, and everything that's based on it came about? People doing exactly the sort of thing I did here. If it makes you feel better, I do intend to my code into our package management system, where everyone here at the organization is free to use it, deploy it, and improve upon it, if they so desire. –  Michael Martinez Oct 30 '13 at 21:32
    
And FYi, it's not a terrible solution. To the contrary it's a very useful tool. When I was searching online for solutions last week, I came across other people asking where they could find this exact functionality. –  Michael Martinez Oct 30 '13 at 21:47

You've already rejected "other people's bash scripts", but this is a pretty common solution -- some creative use of the logger command can follow a file and send its contents elsewhere.
I personally wouldn't do this in a production environment though.


A better option which requires less scripting hackery is using rsyslogd and the text file input module like yoonix mentioned -- This is a pretty decent solution though there is some potential for lost lines during a file rotation, and if you're on a Linux system with rsyslog as your syslog daemon there's not much additional work required.

syslog-ng also supports a file input source with functionality similar to rsyslog's.


IMHO the best solution - albeit one which requires modifying the application generating these logs - is to log to syslog directly. You don't want to be going through intermediary steps, files, etc. -- syslog is the SYStem LOGger, and things that write logs on a Unix platform should be sending them to syslog.
Implementation of this is, unfortunately, left as an exercise for the reader (and application developer) and may not be possible if your developers are nonexistent, lazy, or incompetent....

share|improve this answer
6  
@MichaelMartinez You would modify the rsyslog configuration currently running on the system. You should NOT be running two syslog daemons. Not to be rude, but you need to stop trying to do it wrong*: Every proper solution to this scenario requires administrative (root) actions on the server or modification of the app. You're going to have to face that reality and deal with whatever group within your organization has root on the systems in question, otherwise this question is off-topic (you're trying to circumvent your organization's policies).... –  voretaq7 Oct 23 '13 at 20:57
4  
@Michael All this tells us is that someone is trying to force the wrong team to implement the fix. –  Andrew B Oct 23 '13 at 22:49
3  
@MichaelMartinez imho, that sounds like a pretty quick route to crippling levels of technical debt. –  Sirex Oct 29 '13 at 19:11
3  
I guess. Generally i've found long term that there's no medals in winning self-inflicted battles. When the technical debt gets to the point it impacts the business ironically the people who diligently worked hard to avoid the elephant in the room tend to end up carrying the can, in my experience. So i'd say cover your ass and get someone to agree in writing the downsides of this. –  Sirex Oct 29 '13 at 19:29
1  
@MichaelMartinez RE: "10s of thousands of people", my answer to your meta question still applies; you are a professional, what your company is asking of you is not. Emphasizing the size of your organization is fairly meaningless here; many of us have been around the block long enough to know that the size or (self) importance of a business are no barrier to management mistakes. –  Andrew B Nov 5 '13 at 21:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.