Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm considering setting up our servers to log to a Mongo Database rather than log files. Logs will then be all on one server, queryable, and overall easier to manage.

I'd love to find a solution that will allow all the different processes I have running to write to DB rather than files (or perhaps something to read the files, pass the logs on and truncate the files). I don't want to have to find a different solution for every process if I can avoid it.

So, does anyone know of an existing solution to this problem?

share|improve this question
    
I'm looking got a generic solution that will take log files from any process. However if it helps I'm mainly concerned with Nginx, Gunicorn, Celery, Postgres and syslogs –  Jake Nov 1 '12 at 20:36
add comment

3 Answers

You aren't very clear about which logs (system, web, other?) that you want to send to MongoDB, but since most things can be dumped into syslog one way or another, that is probably the easiest route.

syslog-ng supports logging to MongoDB directly, so anything you can get into syslog would seem to be fairly easy to then push into MongoDB using that solution.

For other potential solutions, take a look at fluentd, and of course there are other options out there for database logging with syslog.

share|improve this answer
add comment

I use Logstash with Kibana. There's a variety of ways to get logs from files and from syslog to go into Logstash. It uses Elasticsearch as the primary backend, but can also inject into Mongodb.

If you work your way through either the centralized or standalone tutorials here, you'll be off to a pretty good start.

share|improve this answer
add comment

MongoDB won't be of much help if all you're looking for is a drop-in solution. You need to plan ahead on what you need in order to get the advantages. In fact, it might make sense to look into RDBMS if dumping log data is all you need as that would allow you to enable compression while MongoDB can in some cases consume more disk space and require more RAM than MySQL if used improperly. Look here for a disk space usage comparison between MySQL and MongoDB: http://blog.trackerbird.com/content/mysql-vs-mongodb-disk-space-usage/

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.