Alternatives to Splunk?

Question

I'm pretty impressed with Splunk, especially version 4. Pretty graphs, alerting (Enterprise only), and fast, accurate, searching. It's a great product.

However, the cost just way too high to consider for full production use for our company. All we really need is to be able to index different logs in a central place, and have reasonable searching on that. Having alerts based on a saved search is also really nice. We don't really go beyond that.

In fact, our biggest usage has been in deploying new applications. Everything gets logged via log4net to either the Event log on Windows or a text file on Linux. Splunk makes it pretty easy to quickly search across those to make sure all the parts of the app are working ok -- that's saved us tons of time versus hunting down individual logging sources.

What alternatives exist in this market? I have a sinking feeling Splunk's pricing is so high because they have the best product by far, and they know it. We want the server to run on Windows.

I'd be open to a split model, using one product for general logs (collect via syslog/Snare), and a dedicated product for our custom apps (like Log4Net Dashboard).

Would using a simple syslog server such as Kiwi, sent to SQL Server (perhaps with fulltext enabled) work?

I'd hope the cost should be well under 5 figures, USD. (And yes, I know, we're cheap. We're a startup with little money, and BizSpark takes care of all our MS licensing.)

Edit: I should add, we have about 10 physical servers, 20 VMs, and a couple firewalls and switches. 90% is Windows.

Answer 1

Note : This is all regarding Linux and free software, as that's what I mostly use, but you should be fine with a syslog client on Windows to send the logs to a Linux syslog server.

Logging to an SQL server: With only ~30 machines, you should be fine with pretty much any centralised syslog-alike and an SQL backend. I use syslog-ng and MySQL on Linux for this very thing.

Pretty frontends for graphing are the main problem -- It seems that there is a lot of hacked-up front-ends which will grab items from the logs and show how many hits, alerts etc but I've not found anything integrated and clean. Admittedly this is the main thing that you're looking for... (If I find anything good then I'll update this section!)

Alerting: I use SEC on a Linux server to find bad things happening in the logs and alert me via various methods. It's incredibly flexible and not as clicky as Splunk. There's a nice tutorial here which guides through a lot of the possible features.

I also use Nagios for graphs of various stats and some alerting which I don't get from the logs (such as when services are down etc). This can be easily customized to add graphs of anything you like. I have added graphs of items such as the number of hits made to an http server, by having the agent use the check_logfiles plugin to count the number of hits in the logs (it saves the position it gets up to for each check period).

Overall, it depends on how much your time will cost to set this up, as there are many options which you can use but they aren't as integrated as Splunk and will probably require more effort to get doing what you want. The Nagios graphs are straightforward to set up but don't give you historical data from before you add the graph, whereas with Splunk (and presumably other front-ends) you can look back at the past logs and graph things you've only just thought of to look at from them.

Note also that the SQL database format and indexing will have a huge effect on the speed of queries, so your idea of fulltext indexing will make a tremendous increase to the speed of searches. I'm not sure if MySQL or PostgreSQL will do something similar.

Edit : MySQL will do fulltext indexing, but only on MyISAM tables prior to MySQL 5.6. In 5.6 Support was added for InnoDB.

Edit: Postgresql can do full text search of course: http://www.postgresql.org/docs/9.0/static/textsearch.html

Answer 2

More aimed at *nix than windows, but octopussy does support windows, and seems to aim at the same kind of thing as splunk.

Answer 3

I'm in the middle of trying out a number of monitoring solutions - but I want to mainly monitor windows. Most of the systems are geared to SNMP monitoring which manage to pull out a remarkable amount of info without agents.

These are some of the systems I've tried so far:

Nagios - Open source. A pig to configure but highly rated and seems very flexible. It seems to be essentially a counter recorder and does not allow for remote script execution and so cannot be used to pick up on configuration problems, ala MS system center or Kaseya. Agentless but is essentially useless without the NSclient tool installed on each client.

Cacti - Pretty and straightforward graphing tool based on pulling snmp stats. Agentless.

OpsView - Based on Nagios but easier to configure and has a better front end.

HypericHQ - Easy to get up and running under Windows. The base version is free and does plenty. There is a commercial HypericHQ enterprise. Agent has to be installed on each client.

Zabbix - Another nice monitoring tool. Its easier to use than nagios. Has an agent you can install on windows and client machines. I've only explored this one a bit so far.

Zenoss - Open source. I have been very impressed by how professional Zenoss is. Its an SNMP based monitor and has loads of extensions to permit the monitoring of HP proliants, windows services, ms sql server, mysql. The extensions all work via SNMP so nothing needs to be installed on the client machines. I haven't explored it all yet and there appears to be much functionality which I have yet to exploit. Its based on Zope so unless you are up to speed on Zope installs I'd recommend downloading the pre-prepared VM - it works like a dream straight out of the box.

On the commercial front you could take a look at a few tools:

Kaseya - costs about 6k per year for 250 nodes , if I remember correctly, but is a superb tool and has a very active user community. Its aimed at the msp market and allows monitoring of multiple companies systems. It can be used internally without problems.

GFI Hounddog - simpler than Kaseya but very cheap at the moment. Definitely worth a look.

There are a number of solutions out there sold as MSP systems but which are essentially monitors + remote admin combined.

Ian

Answer 4

For centralised syslogging with lots of great features I can't help but recommend rsyslog enough. Its an open source syslog server which can happily operate as a drop-in replace for the regular syslogd that you know and love. Its now the syslog daemon of choice for Ubuntu and I think Red Hat & Fedora might be going down that path as well. I've found its alot easier to get up and running and do what you want that syslog-ng is.

Currently in our shop we've got two central rsyslog servers (one in each site) which receives logs for hundreds of servers. I've got automatic email alerts whenever something in syslog triggers alert or higher (with some tweaking of course, some apps are a little bit alarmist). I could probably do some more smarts like getting it to send stuff to nagios or such but it covers us enough for our needs for now.

This all goes into a mysql database as well (there's also support for Oracle or postgresql if that's how you roll).

There's also a web frontend and a windows agent for sending Eventlog logs to the rsyslog server as well. The web frontend obviously isn't as slick as splunk but it gets the job done for $0.

Answer 5

Take a look at http://www.codeplex.com/polymon

Its open source, uses SQL Server at backend and has fancy UI

Answer 6

I agree that Splunk is awesome. For small, dominantly Linux environments, though, you may wish to look at something like epylog.

We used it at one of the places I used to work, and it was great for what we wanted.

Not sure how well it'd handle Windows syslog messages that are sent to a Linux syslog collector, but may be worth a shot.

Answer 7

Just linking to my answer else where:

Splunk is fantastically expensive: What are the alternatives?

Edit (new projects):

The LogStash and Graylog2 projects look very interesting

Here are a couple of Videos: one two.

Answer 8

Something like GFI EventsManager might do the trick for about $4k.

• Analysis of event logs including SNMP Traps, Windows Event logs, W3C logs and Syslog
• Real-time alerts, SNMPv2 traps alerting included
• View reports on key security information happening now
• Centralized event logging
• Remove “noise” or trivial events that make up a large ratio of all security events
• Real-time 24 x 7 x 365 day monitoring and alerting
• Graphically monitor the status of GFI EventsManager and your network through the built-in status monitor
• Support for virtual environments

Answer 9

If you are looking for a SysLog replacement, you might also want to consider a commercial syslog/rsyslog replacement like LogLogic, http://loglogic.com. We (its where I work) have a a full featured logging, storage and reporting set of appliance. Essentially, its the ability to collect 100,000s messages per second, sore and index them so searches can be done.

Answer 10

You could try logscape from liquidlabs - very similar to splunk but has a few different features as well.... http://www.liquidlabs-cloud.com/products/logscape.html

Answer 11

I did the SQL backend thing at a previous job (it was MySQL by the way), complete with scripts, Drupal interface with custom PHP scripts, the works.

Honestly, it took way too much man-hours and still wasn't Splunk.

Currently, I am testing Splunk instead. Yeah, it's not free, but looking at the big picture it might actually be cheaper.

Answer 12

Have you tried php-syslog-ng? http://code.google.com/p/php-syslog-ng/

Answer 13

I posted the dupe thread: Splunk is fantastically expensive: What are the alternatives?

xpolog and all the serious commercial solutions are BIG $ (even if less than splunk, most are easily 5 digits!)

Sooooo, what we finally did (cause splunk was too much $):

1) We wanted a simple syslog to sql db pipeline

2) We tried kiwi syslog. This worked great for a week, stopped working, and kiwi support could not fix it. So we dropped kiwi

3) We tried winsyslog. An old dog of an app, we didn't want to learn it.

4) We used this free .net app: http://www.aonaware.com/syslog.htm

Voila. We have syslog messages in our db.

We are very happy. $0 spent, some hours, but not too much.

Answer 14

We are using Splunk here, and I'm kind of shocked by the pricing they told you. The basic breakdown we were given came in somewhere around $1k US per 1GB of data. Its costly, but super powerful and really fast to develop with. Depending on your data sources and what you want to do with them, some python and perl scripts could give you a lot of similar data. The big difference will be time, and learning to really wield the language for text processing. You also wouldn't be able to get realtime IP information (stuff like syslog), though you can fix this by getting a syslogger and outputting the information to a text file. Sorry I can't point you towards any specific solution; what we can't use splunk for we use python, perl, and bash scripts for.

Answer 15

ELSA - Enterprise Log Search and Archive

Main features:

---

• Full-text search on any word in a message or parsed field.
• Group by any field and produce reports based on results.
• Schedule searches.
• Alert on search hits on new logs.
• Save searches, email saved search results.
• Create incident tickets based on search results (with plugin).
• Complete plugin system for results.
• Export results as permalink or in Excel, PDF, CSV, and HTML.
• Full LDAP integration for permissions.
• Statistics for queries by user and log size and count.
• Fully distributed architecture, can handle n nodes with all queries executing in parallel.
• Compressed archive with better than 10:1 ratio.

---

Performance details:

---

For spec'ing a system, in order of importance: disk size, RAM, disk speed, number CPU's. The overriding performance factor is Sphinx's indexer and search daemon, so refer to sphinxsearch.com for docs. My given stats are taken from large systems (16 CPU, 144 GB RAM, 12 TB HD), but you will get the same performance on a system with 4 CPU, 8 GB RAM, and any sized HD as things scale linearly. The system first ran on IBM blades with 4 GB RAM and slow SAN drives and performed at about the same rate, but 4 GB is cutting it a bit close.

---

Performance details and main features list, plus a description of the architecture: http://ossectools.blogspot.com/2011/03/fighting-apt-with-open-source-software.html

Code: https://code.google.com/p/enterprise-log-search-and-archive/

VM: http://ossectools.blogspot.com/2011/07/elsa-vmware-appliance-available.html

Details concerning the project: http://ossectools.blogspot.com/2011/03/comprehensive-log-collection.html

Answer 16

If you are looking for a much more affordable alternative to Splunk - try LogZilla (http://www.logzilla.pro). It scales as well or better than Splunk (you can search over 300m logs in about 1-2 seconds) and is easily 1/10 of the cost. They have a demo running at http://demo.logzilla.pro