I was just watching the log files for my web server and saw a few log entries for SSHD. I can see that someone is continuously trying to connect via ssh with wrong username and password.
i just saw that by chance.
Aside from sshd logs, what are the other programs/services/logs should I watch for possible hacking attempts.
Linux, ssh, Apache and other basic building blocks of a server are fairly secure, given that you have strong passwords, ssh keys and similar guards in place. It's mandatory to get the basics right.
BUT!
It's much more likely that your application -- such as WordPress, Drupal or your custom piece of software -- running on top of those building blocks has a security hole. Nowadays many hacks are coming through specially-crafted URLs with SQL, strange characters and stuff like that in them. THOSE are the things you should really prepare yourself for. Think of the ways your server might be abused over http and try to stop them before they happen. Even if someone cannot gain root through some web application security hole, there's much abuse which can be done, such as spamming or DoSing, even without the ultimate root access.
For example, it can be a good idea to tighten PHP's security by making your webroot and /tmp (or whatever temp directory you have defined for PHP) noexec,nosuid,nodev so even if someone succeeds in uploading an evil C source file, they can't compile or at least easily run it with things like system("./myevilrootgainer");. That simple precautionary thing among many others is not because PHP itself is insecure (which it sometimes can be, like all software), but it's because it makes exploiting the possible security holes in your application harder.
Basically you should watch any log that corresponds to a services that is available to the network. Particularly if that is open to the entire Internet.
Since the list of services that you may install is huge there is not really any way to give you a single list.
For example if you install a database server like mysql and make it available to the network, then you should watch the mysql logs.
If SSH is open then you should watch your auth logs. Search around on this site for lots of questions that cover using denyhosts, fail2ban, and/or firewalls to limit your exposure to brute force SSH attempts.
You should setup a firewalls to block access to anything on your system except what you actually need to make publicly available. Anything that you make available should be monitored.
Which Services and Porgrams?
All of them.
No really.
You should ensure that any services you do not require are not run. For preference ensure that they can't run.
For any services which should only be accessible locally, then external access should be blocked on the firewall and in the configuration (where possible).
Make sure you implement vendor/distributor security patches promptly.
People are usually surprised when I tell them to leave logs till the last thing on the list when setting up security management policies: logs only tell you about legitimate access and failed access and functional issues - none of which are particularly relevant to keeping out the bad guys.
As for network IDS? Same thing - on most internet connected machines you're going to see so many failed attacks that you won't be able to detect let alone respond to the real thing.
Do use host-based IDS (file integrity checker) for any executable files / libs. It doesn't prevent an attack - but if your systems do get compromised at least there's a chance that you'll be able to recover from the attack.
In addition to something like Tripwire or L5, if this system acts as a server on the internet or hosts data of any significant value, I'd recommend running a rootkit checker at regular intervals (e.g. rkhunter)
If your objective is not to keep your system secure, but rather understand better how people attach systems, then that's a different story (but don't try this on a system where there anything valuable). Yes you might get some useful information from the logs (particularly messages, security and httpd/access.log, httpd/error.log).
You may be interested in /var/log/... (names may be different on your Linux flavor, I'm using ubuntu 10.04)
auth.log logs authentications, like gdm login, su, user/groupadd activity, screensaver unlocks, imap/pop3 connections, cron sessions ...
apparmor/* (if apparmor installed) that shows programs strange behaviors (like accessing a directory they are not supposed to)
messages and syslog, basically the general logging files, when no specific log-file was provided.
mail.log (or maillog) to check your smtp (if any) server activity
samba/* (if running) to check who connects to your samba shares
mysql/* (if relevant) to check the logins list
Depending on your distro, you may or may not have the above files. You could do on a regular basis a ls -lrt on your /var/log directory to see which log files changes, and check the relevant ones.
I would just recommend the Open Software "SNORT". The software provides a vantage point of view for Linux syslogs and acts as a IPS against most intrusion attempts.
SNORT is probably not a good option for you at this time until you have more familiarity with the basics and even then the utility of SNORT for a web server is questionable, besides its not an IPS anyway. I would go a WAF route if all the system is doing is serving up HTTP/S application pages. Otherwise, the first answer is your best option at this stage. Become familiar with running processes and services and what your firewall is permitting.
