Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to stream Varnish's log into MongoDB for realtime reporting.

Can this be configured directly within Varnish or can the official Varnish-Agent (https://github.com/varnish/varnish-agent) be used or some other agent or script?

share|improve this question

3 Answers 3

I've also thought about logging statistics into mongoDB since it has a "fire and forget" as default write mode and it would not block requests but never got further with it.

I see 3 possible solutions:

  1. There's no mongoDB vmod available yet (https://www.varnish-cache.org/vmods) and this may be a kickstart for one.

  2. Use the mongoDB C lib (http://www.mongodb.org/display/DOCS/C+Language+Center) and write a inline C inside your VCL using the runtime functions (VRT_GetHdr and etc).

  3. If you're gonna save few stats as page visits and error codes, you can parse the log from varnishncsa with a python script and store info in mongoDB for latter usage (map and reduce). But this is restricted to a Apache's common log format (there's not much information in there).

share|improve this answer

I decided to use a PHP script with ran a command the varnishstat -1 command via SSH. I based it on this project https://github.com/redsnapper8t8/pyvarnish

share|improve this answer

You don't really want it done in Varnish. What you want is to do have something like varnishncsa talk directly to mongodb. That way if Mongo slows down page delivery isn't slowed down. We're currently playing with idea of building a varnishlogjson program to output (parts of the) varnishlog in JSON format, for insertion into hadoop, but maybe Mongo would be able to eat JSON as well.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.