Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm watching the video at http://www.splunk.com and as someone who is newer to IT management this seems like a great solution to get me started. But I have concerns. I just moved from cPanel and I don't want to end up reliant on another heavy bogged down, overloaded system. I'm wondering if any of you use it, if so, what do you like or not like about it?


I'm really looking for a solution to help sort through server logs and diagnose when the server is under attack. Splunk seems like a very good solution, but is there a better one, preferably free out there?

share|improve this question
4  
This is pretty subjective. Is there a way you can rephrase the question? Maybe something specific that you can ask about? –  squillman Aug 16 '10 at 18:12
    
Splunk is free to download and try. It might be worth the time and effort it would require to do that. As squillman said, I would try to reword your question so it doesn't get closed as subjective –  GregD Aug 16 '10 at 22:11
add comment

5 Answers

up vote 2 down vote accepted

Install the logcheck package. It will scan the logs once an hour and email you anything it doesn't consider normal. Essentially, it emails anything that entered the logs in the last hour that it doesn't have a rule for ignoring. There are additional attack rules than include things which shouldn't be in the log. The email subject line varies depending on the reason things were picked up.

I generally build a local ignore file for it as I discover things which I consider normal, but don't have existing ignore rules.

The various syslog alternatives all support server consolidation, so you can forward the logs to a single server. However, I haven't been in the habit of doing it. The only system I forward logs off of is my OpenWRT firewall.

EDIT: I do use Splunk at work to search log files, although if I known the particular log I am looking for I am more likely to use less. It does have alert capabilities, but we don't use them. I expect they would alert on a match to a known record. This can lead to a lot of false negatives if you have new problems without an alert rule. I prefer to have false positives like I get from logcheck. Splunk may have better timeliness on alerts though.

I do get timely alerts from fail2ban on cases that cause it to trigger. It also maintains blacklist entries for the originating source.

share|improve this answer
add comment

We use Splunk for that kind of thing... Windows and Linux hosts all forward their logs to Splunk and there's some "saved searches" in splunk that generate alerts. Makes it possible to detect things like "more than X failed logins in the last 30 minutes" across all usernames and systems (the saved search is complex, though, in order to capture windows, linux system and various applications...). It's also great for searching through logs right now.

Splunk can be free for a small enough dataset.

You do need to scale the server appropriately, based on how much data you're sending to Splunk per day, how much you want to retain and how much searching you'll be doing. Otherwise it can bog down.

Before Splunk we used to use SEC on a syslog server that had all the logs forwarded to it. Much harder to write saved searches for, and doesn't really do anything to replace grep for searching for things after the fact. Still pretty decent, though.

share|improve this answer
    
Thanks for the feedback. Using Splunk, you would recommend it? –  Webnet Aug 16 '10 at 19:31
    
I'd recommend it to somebody who frequently needed to do the things it's good at, and had a budget where it wasn't difficult to afford... –  freiheit Aug 16 '10 at 20:54
add comment

Splunk isn't really in the same category of software as cPanel. From what I recall cPanel is a web based system management package. Splunk is a data analysis and alerting tool.

That said, according to our website:

"Download Splunk for free. You'll get all of the Enterprise features of Splunk for 60 days and you can index up to 500 megabytes of data per day. After 60 days, or anytime before then, you can convert to a perpetual Free license or purchase an Enterprise license to continue using the expanded functionality designed for multi-user Enterprise deployments. "

Sufficed to say, you can download and use Splunk, for free, on most average sized system log data sets.

As for diagnosing when your server is under attack, that's where Splunk would meet your needs. Check out my blog post from today where I show you how to setup iPhone alerts when someone attempts to login to your server with invalid credentials.

http://blogs.splunk.com/2010/08/16/how-to-use-notifo-to-receive-splunk-alerts-on-your-iphone/

Feel free to ping me back if you have any further questions or would like an extended demonstration license of our product.

share|improve this answer
add comment

One other thing to add. Our company recently looked into purchasing Splunk. We definitely had more than 500MB of logs to analyze and we found that their licensing model was outrageously expensive. Splunk has taken advantage of their increase in popularity and slowly increased their prices over the years. When we first looked at it 2 years ago, the limit on free was 1GB and the licensing fees were half of what they are now.

Splunk is a fantastic tool, but at it's current price, I would think hard about alternatives IMHO.

share|improve this answer
    
I'm in the same boat. With 2 - 3 heavily used production servers and a handful of VMs for specific applications, Splunk's licensing model becomes expensive very quickly. –  Josh Aug 17 '10 at 18:58
add comment

I use and highly recommend http://papertrailapp.com for remote syslog. After you build your alerts and search filters its amazing and inexpensive!

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.