6

My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own file-based database(FSFS) and not plain text file,I have two options

  1. Run svn log via cron(on 1 min interval) and then send that file to logstash(really bad idea)
  2. Used subversion river plugin,I tried it and its not working as it not able to index the data at all.In between its development is stop almost 1 year ago.So no help at all

I also thought about post commit script that whenever any user checkin it will trigger it and then store the logs in text file,but its same as Point 1.

Any help/idea to do that is really appreciated

EDIT

We write this small post-commit as I mentioned in point 3 so that whenever user check-in we can save the metadata in some file and then via syslog we can transfer this log to logstash server.One of the biggest drawback of this approach is I am dealing with TB of data and 15+ checkin per min,this file goes really big(we can use logrotate) but at the same time facing locking condition issue(as multiple user try to check-in and writing to the same file) which will eventually lead to race condition and make the situation more worst.Pasting post-commit hook below so that it might be useful for other people

 #!/bin/sh

 REPOS="$1"
 REV="$2"

 LOG="/tmp/svn.log"

 var1=/usr/bin/svnlook info -r $REV $REPOS | tr '\n' '|'`
 var2=/usr/bin/svnlook changed -r $REV $REPOS | tr '\n' ' '`
 echo "r${REV}|${var1}|${var2}\n" | tee -a ${LOG} 2>&1
 echo " " | tee -a ${LOG} 2>&1
4
  • Why would number 1 be a bad idea? Because it generates so much info? In that case your third option would be a nice option? Hook on the commit, so only relevant info is sent to logstash? – Tim Stoop Feb 23 '15 at 14:52
  • Thanks Tim,Regarding option 1,that is correct as I am dealing with TB of data,unfortunately option 3 is also rejected so I am looking for some new options – Prashant Lakhera Feb 23 '15 at 18:05
  • Can you explain why option 3 was rejected? – Tim Stoop Feb 24 '15 at 18:31
  • The reason is I am dealing with 2TB of data..with 700+ repository and atleast 15+ checkin/min,so to maintain that big file as well maintain concurrency is really difficult – Prashant Lakhera Feb 25 '15 at 4:44
2

I see at least one convenient option: 1) Feed you SVN logs to syslog, most distributions use rsyslog now, so here is example for rsyslog (5.x) :

$InputFileName /${path_to}/svn.log
$InputFileTag svn:
$InputFileStateFile /var/spool/rsyslog/svn_log

$InputFileSeverity notice
$InputFileFacility local7
$InputRunFileMonitor

:syslogtag, isequal, "svn:" @@${IP_of_logstash}:$PORT
&~

Please note, that config will different for newer versions of rsyslog. version 8.x config:

#reading SVN logs
input(type="imfile" File="/var/log/${path_to}/svn.log"
Tag="svn:"
StateFile="/var/spool/rsyslog/svn_log"
Severity="normal"
Facility="local7")

:syslogtag, isequal, "svn:" @@${IP_of_logstash}:$PORT
&~

2) Configure logstash syslog listener and parser for logs

In this case logs will not be stored additionally in syslog, but forwarded directly to logstash and syslog transport will take care of it.

0

One possibility could be Configuring Syslog to accept the subversion logs and send it to logstash.

usefull links

http://logstash.net/docs/1.1.9/outputs/syslog

http://linux.die.net/man/5/syslog.conf

http://www.commandlinefu.com/commands/view/11687/send-apache-log-to-syslog-ng

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.