1

I'm draining various logs from Heroku to an rsyslog linux (ubuntu) server and am starting to have a little more to bite off than I can chew in terms of working with my log histories.

I am needing to be able drill back in time based on more flexible details and more flexible access than what the standard syslog file(s) provide.

I'm thinking that logging to mysql may be the correct approach, but how do I set this up such that it pulls only certain log entries into a table based on an identified? For example, I see a long hex string identifying each log entry from a certain Heroku app instance. I assume that I can just pipe those into the mysql socket vs ALL rsyslog input into mysql...

Could someone please direct me to a resource that can walk me through the process of setting something like this up or simply provide some details that can help? I have 15+ years of Unix experience so I just need some nudging in the right direction as I've not really done a tremendous amount of work with syslog daemons previously in terms of pooling various servers into one.

Additionally, I'd be interested in any log review tools that could make drilling through log arrangements like this more handy for developers.

2 Answers 2

3

I guess something along this should work. First, make sure you have the line below in your rsyslog conf file, so MySQL output module gets loaded:

$ModLoad ommysql

Then, filter the stuff you want like this:

:msg, contains, "Heroku"     :ommysql:dbserver,dbname,dbuser,dbpass;heroku-template

Next you need to create a template for your data, so rsyslog knows what and how to insert to your database. Here is an example template which only records date and message, and is excepting you to have a table with only two columns, date and message.

$template heroku-template, "INSERT INTO herokulog(date,message) VALUES ('%timestamp','%msg')\r\n", SQL

Something like this should work. Note that this was on the top of my head and not tested at all, but you get the idea. Much more detailed information can be found from rsyslog web site, for example the available template parameter values might interest you.

3
  • Okay, this appears to nearly have it. Thought I had it solved, but then realized msgs were not going to the location to which I was attempting to direct them. I stepped back and decided I'd just push them into a /var/log/heroku_test.log file first. I get lines like Nov 20 07:43:10 d.fb592b21-556a-4cbf-bff3-7657231b0c63 app[worker.1] ** Invoke environment (first_time) -- I attempted to simply put in a :msg, contains, "d.fb592b21-556a-4cbf-bff3-7657231b0c63" /var/log/heroku_test.log into the 50-default.conf for rsyslog.d and while the file's created in /var/log, nothing is pushed into it.
    – ylluminate
    Nov 20, 2011 at 7:47
  • I actually ended up getting it to work by working doing something along these lines: :hostname, contains, "d.fb592b21-556a-4cbf-bff3-7657231b0c63" -/var/log/heroku/server_name.log;HerokuFileFormat and having set up a template for the output so as to eliminate the long hostname from that particular server: $template HerokuFileFormat,"%TIMESTAMP% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" - Now that this is working I'll push it to MySQL. Thanks for the push in the right direction.
    – ylluminate
    Nov 21, 2011 at 7:10
  • Glad I was able to help you! :) Nov 21, 2011 at 7:41
1

I'd pipe the syslog to a grep command to extract the lines you need, then to an awk script to generate a well formated csv file then I'd pull the generated csv file into mysql using " LOAD DATA IN FILE "

You cans also check the phpsyslog-ng project : http://www.gdd.net/index.php/PHP-Syslog-NG

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .