Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have just setup a Graylog2 server and I am looking to send all logs from my main server to the graylog server. I have enabled logging for the main server and am sending logs to my graylog server by adding *.* @logs.example.com:1337 to /etc/rsyslog.conf.

What I want is to have Graylog2 collect all my Apache logs, system logs (for SSH logins, rejected logins) and any other logs I need to monitor.

For the Apache logs, I would also like the Rails logs. My sites are located in /srv/www/ and then the structure is sitename.com/public_html and sitename.com/logs. I have many sites on the server and I would like an easy way to view all of the errors and make some nice graphs out of them hence why I want to use Graylog2...

The log files in the logs folder are access.log and error.log.

The Rails logs would be in sitename.com/public_html/log. This contains production.log.

share|improve this question
    
What is the question here? If you have rsyslog installed you can use it to send the Rails/Apache logs to Graylog2 (rsyslog.com/doc/imfile.html) –  polynomial Sep 13 '11 at 6:12

4 Answers 4

Graylog2 only accepts logs in two formats: standard syslog and the Graylog extended log format (aka GELF). Arbitrary logs on disk are going to need some third-party process to consume the logs and translate it into a form that's useful for you.

Take a look at Logstash. Most people think of it as a tool to index logfiles using ElasticSearch, but it also contains a general purpose "log router" that lets you tail a bunch of files on disk and send them to a logging component like Graylog.

share|improve this answer
2  
Rsyslog can easily read the logs: rsyslog.com/doc/imfile.html –  polynomial Sep 13 '11 at 6:11
1  
While rsyslog can use other files as input, using logstash has the advantage of being able to use the predefined regular expressions of Grok (see logstash.net/docs/1.0.17/filters/grok) to get some structure into the log messages and not just plaintext. –  joschi Dec 17 '11 at 19:08
    
syslog-ng can do the same with cleaner config compare to rsyslog: loggly.com/support/sending-data/logging-from/syslog/syslog-ng/… –  HVNSweeting Mar 28 '13 at 3:28

This is old, but I thought I would write this method which I use for low/medium traffic site (don't know if it will work well for heavy traffic site):

In Apache, I define a CustomLog format called graylog2_access which formats the access log into a GELF format and then I send my log through Graylog2 by piping the log data through nc to send GELF messages to Graylog2's input.

Here is the custom format that it creates (human readable):

{ 
 "version": "1.1",
 "host": "%V",
 "short_message": "%r",
 "timestamp": %{%s}t,
 "level": 6,
 "_user_agent": "%{User-Agent}i",
 "_source_ip": "%a",
 "_duration_usec": %D,
 "_duration_sec": %T,
 "_request_size_byte": %O,
 "_http_status": %s,
 "_http_request_path": "%U",
 "_http_request": "%U%q",
 "_http_method": "%m",
 "_http_referer": "%{Referer}i"
}

For the Apache config, here is a copy/paste version:

LogFormat "{ \"version\": \"1.1\", \"host\": \"%V\", \"short_message\": \"%r\", \"timestamp\": %{%s}t, \"level\": 6, \"_user_agent\": \"%{User-Agent}i\", \"_source_ip\": \"%a\", \"_duration_usec\": %D, \"_duration_sec\": %T, \"_request_size_byte\": %O, \"_http_status\": %s, \"_http_request_path\": \"%U\", \"_http_request\": \"%U%q\", \"_http_method\": \"%m\", \"_http_referer\": \"%{Referer}i\" }" graylog2_access

Then in your host configuration:

CustomLog "|nc -u graylogserver 12201" graylog2_access
share|improve this answer
    
+1 Nice one! Have to try this on one of our systems, but with ERROR instead. –  Henk Mar 22 at 16:28
    
@Henk - if you do a format for Error log, please let me know, I want to but haven't taken the time yet. Also, check serverfault.com/questions/582510/… for some info of things I ran into later on. –  ETL Mar 22 at 19:01

You can also send your log files to graylog2 server using this simple command:

tail -F -q $yourlogfile |   while read -r line ; do   echo "<7> $hostnamesendingthelog $line" | nc -w 1 -u $graylogserver 514;   done;

I use this mainly for test purposes to determine if my log format is adapted for easy querying in graylog2. For production use you will wan't to set up rsyslog or syslog-ng.

You can probably tail your rails log file and see what happens.

share|improve this answer

You can use the apache2gelf scripts from here.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.