11

I have just setup a Graylog2 server and I am looking to send all logs from my main server to the graylog server. I have enabled logging for the main server and am sending logs to my graylog server by adding *.* @logs.example.com:1337 to /etc/rsyslog.conf.

What I want is to have Graylog2 collect all my Apache logs, system logs (for SSH logins, rejected logins) and any other logs I need to monitor.

For the Apache logs, I would also like the Rails logs. My sites are located in /srv/www/ and then the structure is sitename.com/public_html and sitename.com/logs. I have many sites on the server and I would like an easy way to view all of the errors and make some nice graphs out of them hence why I want to use Graylog2...

The log files in the logs folder are access.log and error.log.

The Rails logs would be in sitename.com/public_html/log. This contains production.log.

10

This is old, but I thought I would write this method which I use for low/medium traffic site (don't know if it will work well for heavy traffic site):

In Apache, I define a CustomLog format called graylog2_access which formats the access log into a GELF format and then I send my log through Graylog2 by piping the log data through nc to send GELF messages to Graylog2's input.

Here is the custom format that it creates (human readable):

{ 
 "version": "1.1",
 "host": "%V",
 "short_message": "%r",
 "timestamp": %{%s}t,
 "level": 6,
 "_user_agent": "%{User-Agent}i",
 "_source_ip": "%a",
 "_duration_usec": %D,
 "_duration_sec": %T,
 "_request_size_byte": %O,
 "_http_status": %s,
 "_http_request_path": "%U",
 "_http_request": "%U%q",
 "_http_method": "%m",
 "_http_referer": "%{Referer}i"
}

For the Apache config, here is a copy/paste version:

LogFormat "{ \"version\": \"1.1\", \"host\": \"%V\", \"short_message\": \"%r\", \"timestamp\": %{%s}t, \"level\": 6, \"_user_agent\": \"%{User-Agent}i\", \"_source_ip\": \"%a\", \"_duration_usec\": %D, \"_duration_sec\": %T, \"_request_size_byte\": %O, \"_http_status\": %s, \"_http_request_path\": \"%U\", \"_http_request\": \"%U%q\", \"_http_method\": \"%m\", \"_http_referer\": \"%{Referer}i\" }" graylog2_access

Then in your host configuration:

CustomLog "|nc -u graylogserver 12201" graylog2_access
  • +1 Nice one! Have to try this on one of our systems, but with ERROR instead. – Henk Mar 22 '14 at 16:28
  • @Henk - if you do a format for Error log, please let me know, I want to but haven't taken the time yet. Also, check serverfault.com/questions/582510/… for some info of things I ran into later on. – ETL Mar 22 '14 at 19:01
5

You can also send your log files to graylog2 server using this simple command:

tail -F -q $yourlogfile |   while read -r line ; do   echo "<7> $hostnamesendingthelog $line" | nc -w 1 -u $graylogserver 514;   done;

I use this mainly for test purposes to determine if my log format is adapted for easy querying in graylog2. For production use you will wan't to set up rsyslog or syslog-ng.

You can probably tail your rails log file and see what happens.

  • I like the simplicity of this reply: Straightforward and no extra tools. I don't know what to do about the Log File timestamps: maybe nothing's to be done and they don't matter? – texas-bronius Oct 26 '17 at 14:06
2

Graylog2 only accepts logs in two formats: standard syslog and the Graylog extended log format (aka GELF). Arbitrary logs on disk are going to need some third-party process to consume the logs and translate it into a form that's useful for you.

Take a look at Logstash. Most people think of it as a tool to index logfiles using ElasticSearch, but it also contains a general purpose "log router" that lets you tail a bunch of files on disk and send them to a logging component like Graylog.

2

You can use the apache2gelf scripts from here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.