11

I recently read an article about analyzing malicious SSH login attempts. This got me thinking, are the SSH username, password combinations on my Debian box that uncommon? Had I been targeted by a brute force dictionary attack? Let's take a look at /var/log/auth.log.0:

Sep 23 07:42:04 SLUG sshd[8303]: Invalid user tyjuan from 210.168.200.190
Sep 23 07:42:09 SLUG sshd[8305]: Invalid user tykeedra from 210.168.200.190
Sep 23 07:42:14 SLUG sshd[8307]: Invalid user tykeem from 210.168.200.190
Sep 23 07:42:19 SLUG sshd[8309]: Invalid user tykeshia from 210.168.200.190
Sep 23 07:42:25 SLUG sshd[8311]: Invalid user tyla from 210.168.200.190
Sep 23 07:42:30 SLUG sshd[8313]: Invalid user tylan from 210.168.200.190
Sep 23 07:42:35 SLUG sshd[8315]: Invalid user tylar from 210.168.200.190
Sep 23 07:42:40 SLUG sshd[8317]: Invalid user tyler from 210.168.200.190
Sep 23 07:42:45 SLUG sshd[8319]: Invalid user tylerfrank from 210.168.200.190
Sep 23 07:42:50 SLUG sshd[8321]: Invalid user tyliah from 210.168.200.190
Sep 23 07:42:55 SLUG sshd[8323]: Invalid user tylor from 210.168.200.190

So that doesn't look good. Now that I know I've been targeted by an attack and that some of my username, password combinations are weak, I'd like to know how can I...

  • ... determine if my Linux box has been infiltrated?
  • ... undo any of the damage left by the perpetrators?
  • ... prevent this from happening in the future?

UPDATE

Any advice on undo any of the damage left by the perpetrators?

Improve this question
2
  • that log doesn't suggest you have been compromised. do you have further information beyond that that worries you?
    – Owen
    Commented Sep 24, 2008 at 21:56
  • possible duplicate of How do I deal with a compromised server?
    – Dennis Nolte
    Commented Jul 31, 2014 at 14:00

20 Answers 20

Reset to default
15

A lot of people seem to suggest DenyHosts, but I've seen a lot of success with Fail2Ban on my systems. It watches for a (configurable) number of failures, and then performs an action - on my servers, that action is to use iptables to drop all traffic from the host. After 10 login failures, they get banned and that's the end of it.

I use that in combination with Logcheck, so that I always know what's going on on my servers.

If you have any evidence that someone has actually broken into your systems (the logs you have posted are not evidence of this), then your only solution is to back up all the data you need to keep, wipe the machine, reinstall, and restore from backups. Otherwise, there's no way to be sure.

Improve this answer
3
  • 1
    I use logcheck and fail2ban also. Very good combination.
    – Dave Sherohman
    Commented Sep 25, 2008 at 3:29
  • I second the use of fail2ban.
    – Malfist
    Commented Sep 18, 2009 at 17:40
  • I also switched to fail2ban from denyhosts because fail2ban monitors more services (mail, web, ftp, ...) and is more configurable
    – Jure1873
    Commented Sep 24, 2009 at 19:01
10

Valid login attempts are logged in as well, so if you see a brute force attempt followed by a success, that's a good indication something bad has happened.

I use DenyHosts to monitor my logs for suspicious SSH traffic, and I have it configured to automatically firewall off hosts at a certain point.

Note there are a variety of other ways you'd want to monitor your machine to see if it's compromised, including load patterns, login activity, periodic traffic sniffing, monitoring running processes and open ports, and ensuring file integrity with a tool like tripwire.

If you're only going to do one, monitoring system load is a very effective way of detecting compromise, because most machines when compromised are used to do things like send massive amounts of spam or otherwise receive a lot of traffic. Perhaps not useful if you're a high-value target and people may be trying to specifically break into you for reasons other than to turn your host into a zombie, but valuable nonetheless. Plus monitoring load is needed for profiling and to figure out when you need to invest in more hardware or better software.

You should also do comprehensive log analysis, looking at auth.log and others for things that are unexpected. Log file analysis is a competitive market and the problem isn't yet solved, but there are free tools like logwatch which can be configured to send you summaries daily.

Security through layers!

Improve this answer
3
  • 1
    Of course, the attacker could have modified the logs to remove evidence of their intrusion, so lack of evidence in the logs doesn't necessarily mean everything is okay.
    – sk
    Commented Sep 24, 2008 at 21:50
  • This is true. Lack of evidence is basically never evidence of no compromise, in my book. Logging to a remote server can increase the trustworthiness of the logs. I use syslog-ng and ssh tunnels for this purpose.
    – Daniel Papasian
    Commented Sep 24, 2008 at 22:20
  • I had been seeing ridiculous numbers of attempts to hack my server as well...I installed DenyHosts and it had already added a couple IPs after only like 10 minutes. Thanks!
    – Aaron Brown
    Commented Sep 24, 2009 at 22:35
4

Forget Tripwire, its quite expensive. Use AIDE instead. Its free, easy to setup (though it takes a little while to decide which temp directories to exclude, and otherwise configure).

you run it, it builds a database of all files. Run it again and it'll tell you which files have changed.

One other thing to do is install CSF, which has a denyhost type blocker, as people fail repeatedly to login, it'll add them to your firewall rules. You can also require SSH logins to have a public key as well, the script kiddies can attempt as many logins as they like then.

Improve this answer
1
4 
"* ... determine if my Linux box has been infiltrated?"
  • look for signs of strange processes. I normally use the tools that come with chkrootkit (http://www.chkrootkit.org)
  • Do a portscan with nmap from a different machine. If your box has been compromised, chances are the atacker installed a backdoor

"* ... undo any of the damage left by the perpetrators?"

forget it, if there was an attack, the best advice is to reinstall it from scratch ( make sure you plugin any holes on the new install ). It is very easy to not notice a backdoor or a stealth process, you are better off reinstalling.

"* ... prevent this from happening in the future?"

  • security updates

  • tight firewall

  • strong passwords

  • turn off uneccessary services

Improve this answer
3

take a look at tools like logcheck, portsentry, and tripwire. it's very common for random dictionary SSH attempts, so i wouldn't be too worried by that. you may want to change the port for random obfuscation, but you'll still see random attempts from time to time, it's life having a machine on the internet.

Improve this answer
3

One thing I use on my servers to help prevent these attacks is DenyHosts. DenyHosts will stop a malicious user from attempting to login. Since installing it my log files have had much fewer login attempt entries.

Improve this answer
3

It's a good practice to use Public/Private key pairs as extra authentication; That way a user can't login over SSH without a right key; wich would be pretty impossible to guess for a brute forcer. A nice article about this can be found here.

This alone with a passphrase would be pretty solid for SSH authentication; But there are more vulnerability's! Watch out with all applications that use a open port; If they contain a bug exploiters might overtake your system. A good example was a spam bot that installed on our server due to a bug in the webstats software we were currently using.

Improve this answer
2

Fail2ban is a access log real time analiser. It can be configured to block any IP with a number of failed attempts to log in. This avoids dictionary attacks without having to move the ssh port. chkrootkit and rootkithunter are good utilities to check for intrusion. If intrusion has been successful, best practice is copy data (and just data, not executables),wipe and reinstall because it is really difficult to be 100% sure the system is clean.

Improve this answer
2

There's nothing here to suggest that your box has been compromised. Even if your passwords are pretty weak, a dictionary attack that only throws one login attempt at each username is extremely unlikely to succeed.

But, if you know your passwords are weak, then strengthen them! I personally like pwgen (which is packaged by Debian). On each run, it generates a large number of strong, but relatively-pronouncable password candidates which (for me at least) are fairly easy to remember, such as yodieCh1, Tai2daci, or Chohcah9.

However, if there is other evidence to show that your system has been compromised... Nuke it from orbit. It's the only way to be sure.

Non-executable data is probably salvagable, but anything with executable content (this potentially includes such things as MS Office documents and some configuration files) has got to go unless you're willing and able to either manually examine it to ensure that it hasn't been turned hostile or to accept the possibility that it could be hostile and either damage your system or provide an avenue for a future re-compromise if you keep it around.

Improve this answer
2

One small additional protection I like is to rate limit incoming ssh connections to slow down any dictionary attacks or similar. Don't expect this to protect you all by itself, but use this in addition to the suggestions in the other answers including disabling password authentication.

Using iptables:

$ iptables        -A INPUT      -p tcp --dport 22 -m state --state ESTABLISHED,RELATED -j ACCEPT
$ iptables        -A INPUT      -p tcp --dport 22 -m state --state NEW -m limit --limit 3/min --limit-burst 3 -j ACCEPT
$ iptables        -A INPUT      -p tcp --dport 22 -j DROP
Improve this answer
2

There is some poor advice on this thread, such as:

  • using a non-standard port for ssh (wrong!)
  • using some third party security tool (adding an unnecessary dependency/complexity)
  • configuring your firewall to block or whitelisting (maintenance head-ache)

Simply tweak your /etc/ssh/sshd_config instead to improve security:

  • PermitRootLogin no
  • Configure AllowUsers for only the users on the system who have ssh accounts
  • Consider using only physical keys with 'PasswordAuthentication no'

If you are box has been infiltrated. Rebuild the box.

Improve this answer
1
  • (-1) Configuring rules on the firewall, or using an intrusion prevention system is sometimes necessary. Calling it poor advice without any additional discussion about why it isn't a good idea isn't very helpful.
    – Zoredache
    Commented Sep 24, 2009 at 20:13
1

Will happen all the time with ssh enabled. Move it to a high port.

There is a program called "Tripwire" that is excellent for intrusion detection, but pretty tough to install. If nothing else, you should read through their docs so that you understand the issues.

Improve this answer
1

You need to install intrusion detection before you hook the machine up to the internet.

And it's a good idea to IP whitelist SSH connections just to be sure the whole world can't even try things like that.

Improve this answer
1
  • I would also add whitelisting users allowed to log in via SSH as well as disabling password-based authentication if possible.
    – Alexander
    Commented Sep 24, 2008 at 21:48
1

How can I determine if my Linux box has been infiltrated?

Boot off read-only media (livecd) and compare the files to your backup or the original media.

Just look around for odd behavior. Of course this is easiest if you take the time before a hack to get a good feeling of what is 'normal'.

The errors you posted do not indicate a compromise. Just that someone is trying.

How can I undo any of the damage left by the perpetrators?

Reinstall and restore from a backup before the system was compromised.

See:

How can I prevent this from happening in the future?

See:

Improve this answer
0

You can watch the box to see what's going in and out and look for anything suspicious. See this post: https://stackoverflow.com/questions/124745/sniffing-network-traffic-for-signs-of-virusesspyware

Improve this answer
0

Psad coupled with Shorewall is a good way to compliment your iptables rules.

I also use Fail2ban to track my ssh logins

Improve this answer
0

use rkhunter or chkrootkit or both; scan your box from outside to see what ports are opened

anyhow, if all you have is invalid user, there is no need to worry :)

Improve this answer
0

Something very different: try using Google on the IP address! A hacker who calls himself STARTURK from Turkey might have attempted to hack your site.

While it looks like a brute-force attack on your system, it could well be that this hacker just did one try and is now gone to some other site.

Improve this answer
0

As many have noted, Tripwire/AIDE are the best way to look for system changes. Unfortunately, the cow's out of the barn on that one since it has to be configured on a known-good system.

One thing that might help you at least get started is to use your RPM database to check the md5sums of your files. The basic gist is this:

rpm -qa | xargs rpm -V

This is not perfect for a variety of reasons. First, your local RPM database could theoretically have been altered. Secondly, most distros use prelinking, and RPM isn't prelink aware. The MD5s may appear to have been altered from that process, which is legit.

The best advice is this: if you're not sure if you've been compromised, it's really time for a rebuild.

Improve this answer
0

To prevent future security issues, you can have a look at OSSEC, I use it to do file integrity checks and log monitoring on our servers, it's very complete and easy to configure. It can send mail notification, you can check alerts via command line or a web interface ...

http://www.ossec.net/

taken from the website :

"OSSEC is an Open Source Host-based Intrusion Detection System. It performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response."

  • log analysis It can check logs file on your servers and alerts you via rules (there are a lot pre-defined and you can add your own)

  • file integrity tripwire/aide like functionnality so you will see if any file has been modified on your server

  • policy monitoring : check some "Best practices" security rules

  • rootkit detection : rkhunter, chkrootkit like functionnality

  • real-time alerting and active response : You can configure ossec to react automatically to alerts (i don't use this but you can use it to block ssh access to hosts making too many failed connections attempts)

Really good product and it is very active

To harden your box you can also use lynis or bastille

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .