196
votes

I just checked my server's /var/log/auth.log and found that I'm getting over 500 failed password/break-in attempt notifications per day! My site is small, and its URL is obscure. Is this normal? Should I be taking any measures?

7
  • 2
    Until we locked down all unnecessary external ports, I remember not only did we get lots of hack attempts, but one day it was so bad that we were being hacked from two different countries -- at the same time! So yes, 100s of break-in attempts is perfectly normal. Mar 8, 2011 at 14:53
  • 91
    We have servers that experience a new attack "sequence" once every 16 seconds. A single sequence is usually a batch of around 100 attempts across various ports. Just for kicks one day I turned on an unpatched server outside our firewall; it tooks less than 10 minutes from the time it was powered on for it to get pwnd. Point is the internet truly is a jungle; try not to get eaten.
    – NotMe
    Mar 8, 2011 at 18:55
  • 2
    I can see I posted my question to the wrong site: superuser.com/questions/200896/…
    – Justin C
    Mar 9, 2011 at 22:01
  • 6
    while I agree with others this is normal on common ports required (80, 443) I practically eliminated these attempts against my SSH port by simply changing the default port from 22 to something obscure like 6022 for example. Just doing that, alone, nearly eliminated 99% of that type of attack.
    – Kilo
    Mar 11, 2011 at 20:32
  • 2
    If you're going to change your SSH port, there are security reasons to keep it below port 1024 (only root can open ports < 1024, so it protects you from other users hijacking SSH). Mar 14, 2011 at 17:54

22 Answers 22

207
votes

In today's internet this is quite normal sadly. There are hordes of botnets trying to login to each server they find in whole IP networks. Typically, they use simple dictionary attacks on well-known accounts (like root or certain applications accounts).

The attack targets are not found via Google or DNS entries, but the attackers just try every IP address in a certain subnet (e.g. of known root-server hosting companies). So it doesn't matter that your URL (hence the DNS entry) is rather obscure.

That's why it is so important to:

  • disallow root-login in SSH (howto)
  • use strong passwords everywhere (also in your web applications)
  • for SSH, use public-key authentication if possible and disable password-auth completely (howto)

Additionally, you can install fail2ban which will scan the authlog and if it finds a certain amount of failed login attempts from an IP, it will proceed to add that IP to /etc/hosts.deny or iptables/netfilter in order to lock out the attacker for a few minutes.

In addition to the SSH attacks, it is also becoming common to scan your webserver for vulnerable web-applications (some blogging apps, CMSs, phpmyadmin, etc.). So make sure to keep those up-to-date and securely configured too!

19
  • 21
    Applications such as fail2ban can help a lot to 'temporarily' stop those bots from hitting your server at silly times in the morning :-) I have mine set up to ban 3 incorrect attempts for 24hours.
    – emtunc
    Mar 8, 2011 at 13:06
  • 46
    And move ssh's port from 22 to 222. That works quite well. Mar 8, 2011 at 13:47
  • 40
    +1, public-key authentication only :) Mar 8, 2011 at 14:09
  • 3
    @STATUS_ACCESS_DENIED: the actions fail2ban takes are just lists of shell commands to run. So it's really flexible and easy to make work properly with any custom config. The best reference is to download it and look at action.d/iptables.conf.
    – mattdm
    Mar 8, 2011 at 18:01
  • 4
    Blocking attackers like this is a waste of time. If you disable root login, there's a good chance that no one will ever even guess your correct login name, let alone password. SSH itself is already rate limiting password requests, so even if they know your user name (random bots won't), if you have a decent password, they'll never guess it. Mar 9, 2011 at 19:44
58
votes

A few 100 is just fine... Last month I found one of my servers had 40k failed attempts. I went trough the trouble of plotting them: Map

Once I changed the ssh port and implemented Port Knocking, the number dropped to 0 :-)

9
  • 2
    Nice map. I would love to know how to do this!
    – jftuga
    Mar 8, 2011 at 15:27
  • 9
    @jftuga I got all the IP's from the logs first. grep 'Failed password' /var/log/secure* | grep sshd | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' | sort | uniq(remove the | uniq at the end if you want to allow duplicates). You can then put them in a CSV and upload it to zeemaps.com. I've seen better maps then mine, where they would use the count to color the map (green to red for the number of attempts per county) but I haven't jet figured that one out Mar 8, 2011 at 15:50
  • 3
    What do you mean by 'implemented Port Knocking'? Is there an app i can install via apt-get to do this? The number dropping to 0 sounds nice
    – user274
    Mar 8, 2011 at 21:46
  • 18
    Security through obscurity gets a bad wrap. It's perfectly fine as long it's part of the overall strategy rather than the entire strategy. After all, what else is a password other than an obscure string?
    – Joel Coel
    Mar 9, 2011 at 17:05
  • 5
    @Joel Coel, it's a secret string, as opposed to most security through obscurity issues - an obscure, but not necessarily secret process. Mar 10, 2011 at 6:06
29
votes

I for one use a "tarpit" in addition to only allowing public key authentication and disallowing root logins.

In netfilter there is a recent module, which you can use with (INPUT chain):

iptables -A INPUT -i if0 -p tcp --dport 22 -m state --state NEW -m recent --set --name tarpit --rsource
iptables -A INPUT -i if0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 180 --hitcount 6 --name tarpit --rsource -j DROP
iptables -A INPUT -i if0 -p tcp --dport 22 -j ACCEPT

What that does is, that every attempt to connect to port 22 is listed by the recent module with IP and some other stuff under the name "tarpit" (if you're curious, look at /proc/net/xt_recent/tarpit). Obviously you can use other names.

To list or delist IPs use:

echo "+123.123.123.123" > /proc/net/xt_recent/tarpit
echo "-123.123.123.123" > /proc/net/xt_recent/tarpit

This rate-limits the attempts to 5 in 300 seconds. Please note that users with an existing connection are not bothered by that limit, because they already have an established connection and are allowed to create more (even above the rate limit).

Adjust the rules to your liking but make sure that they be added in that order (i.e. when adding then use them in this order, when inserting then in the reverse order).

This cuts down the noise immensely. It also provides actual security (against brute-forcing) unlike the perceived security of changing the port. However, I'd still recommend changing the port if it's feasible in your environment. It will cut down the noise level a lot as well ...

You can still combine this with fail2ban, although I've been running just fine without it and only the above rules.

EDIT:

It is possible to lock your self out doing this, so you can add something like the following that lets you clear you ban by knocking on a particular port:

iptables -A INPUT -i if0 -p tcp --dport <knockport> -m state --state NEW -m recent --name tarpit --remove
3
  • 2
    I use this and manage to block myself occasionally, so like to set another port up that you can "knock" on to clear your ban.
    – benlumley
    Mar 9, 2011 at 11:46
  • @benlumley: good point. Port knocking can be similarly useful to changing the default port - or even both of them in combination ... Mar 9, 2011 at 12:52
  • @benlumley: saw your comment (now removed by Sam). I absolutely don't mind if the answer gets edited/improved ;) Mar 9, 2011 at 17:36
15
votes

You could implement fail2ban, or similar methods like locking SSH to your IP. Sadly bots attempt to bruteforce access all the time so it is quite normal, you need to make sure you have a good password.

1
  • 3
    If you're using SSH, consider public key authentication. This is a bit more secure than password authentication. Mar 11, 2011 at 12:39
12
votes

Yes. It's quite normal nowadays.

Please use only public key authentication for administrative purposes if possible. Generate a private key on you workstation:

$ ssh-keygen -t dsa

Copypaste the contents of ~/.ssh/id_dsa.pub to you servers ~/.ssh/authorized_keys (and /root/.ssh/authorized_keys, should you require direct root login).

Configure your servers /etc/ssh/sshd_config to only accept public key authentication:

PubkeyAuthentication yes
PasswordAuthentication no
PermitRootLogin without-password

If you have too many servers, you can use Puppet to run public keys and configurations to them.

Look into Denyhosts and fail2ban to block repeated SSH login attempts and see Snort if you require full IDS/IPS.

4
  • 2
    I wouldn't recommend using public key authentication in SSH for shell access to your server. If your workstation is compromised or, even worse stolen, then someone now has open access to your servers without the need for a password. Public key authentication is more for situations where you need something like a script or program to be able to have SSH access to another system without needing a password so you don't have to embed plain-text passwords in your script/program. Mar 12, 2011 at 21:34
  • 5
    @Deleted Account: You can set a passphrase for the SSH private key.
    – Phil Cohen
    Mar 15, 2011 at 22:57
  • 2
    "Registered User's" comment is misguided. Just to clarify: Always set a good password on your private key, and don't store the private key on any servers. Keep the private key on your own workstation. Once you add the key to an ssh-agent program, and enter your password, you can log into every system that has the public key installed, without once having to re-enter your password. Enable agent-forwarding in your ssh client so you can log in from server to server. Getting your private key stolen is bad, but with a decent password on it it's not as bad as a stolen password. May 24, 2011 at 20:47
  • Right, don't even think to store admin's private keys unencrypted.
    – yrk
    Apr 10, 2012 at 9:02
8
votes

use http://denyhosts.sourceforge.net/

and yes, you should use public-key authentication and disable password auth.

1
  • Disabling password auth is not always a viable solution.
    – Publiccert
    Mar 11, 2011 at 19:39
6
votes

The attempts are mechanized, so the numbers seem OK (yes they are high compared to some sites and the low compared to others). You should take the steps you normally have to: You consider your sites as attack targets every day, even when you do not detect an attack; not detecting an attack, does not mean it does not exist.

6
votes

I'd say only getting only 500 is kind of low.

At a previous employer, one of the computer security researchers called the constant stream of break-in attempts the "internet equivalent of cosmic noise". He described it as a normal, continuous flow of malicious traffic that was searching for systems on the internet and automatically exploit scripts to attempt to hijack the system. Bot-nets and other malicious systems would perpetually scan and rescan the internet for vulnerable systems much like SETI.

6
votes

Yes,

this is common, but that doesn't mean you shouldn't fight the good fight. Here are some steps on how you can make your server more secure.

Avoid DNS associated IP addresses

You can greatly reduce this number in shared or colocation environments by disabling SSH access on any IP addresses associated with domain names. Unlisted non-domain IP addresses will receive less of this type of traffic, so buy an unlisted IP and only use this IP for SSH access.

Use a VPN for all SSH access

If you are in an environment in which you can implement IPsec/VPN to a private network within your server environment, this is ideal. Disable all SSH Internet access, make sure you have an integrated lights out solution. Setup your VPN, and only allow SSH access from your VPN.

Implement IP address rules for SSH access

If the VLAN isn't an option configure your router, or firewall rules to only allow SSH connections from a known IP address range.

If you follow these steps you will sleep much easier in the night knowing someone would have to compromise your hosting companies network to gain access to the server through SSH.

5
votes

Pretty normal to see hundreds of failed SSH connections.

If you have the option, I simply change my SSH port to something non-standard. It doesn't necessarily make your server any more secure, but it sure cleans up the logs (and lets you see anyone deliberately trying to break in!)

5
votes

In addition to using an automated lockout mechanism like fail2ban you have one more option: actually contact the abuse address ISP of the attacker. It may seem completely futile but in the case of the script-kiddie their ISP is more than willing to take action on them.

To find the abuse address, start with arin.net and lookup the IP address using whois. You may be redirected to another regional registry but eventually you can find the responsible ISP for the IP block which contains the address. Look for the abuse@ address or just mail the technical contact.

Send them a polite message with the relevant log file entries (make sure to remove any private information) and ask them to take action against the offending host.

4
  • 4
    We used to do this. However the amount of time spent versus the benefit received was so tiny it doesn't matter.
    – NotMe
    Mar 8, 2011 at 18:50
  • 1
    A variant of this tactic which is more effective, but much riskier, is to report the ISP to the intermediate node. But you MUST have your report evidence solid. I got a whole ISP in grave trouble once doing this, because they were ignoring their abuse reports.
    – staticsan
    Mar 10, 2011 at 0:10
  • 1
    One time I did this, the abuse message reached the hacker instead of someone in charge of the servers. Since then, I don't bother anymore, just too much trouble.
    – wump
    Mar 11, 2011 at 15:23
  • This really isn't going to help in most cases and can be time consuming
    – RichVel
    Jun 21, 2012 at 10:16
4
votes

I would recommend not using fail2ban but running SSH (and others) on a non-standard port. I don't believe in security by obscurity but I think that this is an excellent way to reduce the noise in your logs.

Failed logins on non-standard ports will be few and far between and can also indicate more targeted attacks.

You could even go a step further an install a SSH honeypot such as Kippo to 'let in' the bruteforcers and see what they would do given the chance.

1
  • Haha, Kippo looks very nice. I'm going to install it on a server just to see what they are trying to do.
    – wump
    Mar 11, 2011 at 15:29
4
votes

Yes, it's normal. What I tell clients in your situation with small websites.

Always be prepared to be hacked.

Have a copy of your website on a dev server. This can be your Windows desktop using XAMPP which you can get for free.

ALWAYS make changes to your dev server then upload them to your live website. If it is a CMS like Wordpress, make your posts on the dev server then copy and paste them into the live server.

NEVER download anything from your live website to your dev server.

Monitor your webpages regularly for any changes that you did not do. Specifically, hidden links to drugs or 'enhancement' products. You can find lots of browser add ins and programs that will do this for you.

If you are compromised. Notify your host, delete everything, change all passwords and upload your clean dev server to the now empty webserver. Work with your host to prevent a recurrence.

You should not need a security team for a small site. That is what your host is supposed to provide. If they do not, then get another host which is much easier to do when you have a dev server rather than trying to move the live server.

Hope this helps.

1
  • 2
    +1 for "Always be prepared to be hacked."
    – user78940
    May 24, 2011 at 23:05
3
votes

Another way of stopping it (as I personally don't like to move the SSH port): decide if you are able to list all networks from which you'll ever want to login, then only allow these to access your SSH port.

WHOIS entries of local ISPs helped me to reduce attacks to 1-2 login attempts a month (back then it was about 1k/day). I detected those by still using denyhosts.

3
votes

In addition to the other excellent suggestions you've already received, I also like to use the AllowUsers directive if appropriate for the given server. This allows only specified users to login via SSH which greatly reduces the possibility of gaining access through an insecurely configured guest/service/system account.

Example:

AllowUsers admin jsmith jdoe

The option AllowUsers specifies and controls which users can access ssh services. Multiple users can be specified, separated by spaces.

3
votes

Yes, it is normal. You can :

  • Reduce the opportunity for attack by using fwknop

Fwknop is the one of the better port knock implementations because it is not spoofable and actually authenticates as opposed to just authorizing a connection.

  • You can change the port that Openssh uses but you are not really improving security.

  • Fortify ssh authentication using Google-authenticator or wikid

This will protect password based attacks and the possibility of a determined attacker / targeted attack compromising your admin machine and stealing your ssh-key & password combo.

Just look at the latest pwn2own comp to see how easy it is for a skilled attacker to compromise your fully patched admin box.

1
vote

Sadly this is quite normal. You should consider adding something like fail2ban to your system to automatically detect and ban the attackers. If you don't already do so you should also consider only using ssh with public keys and don't allow root login over ssh. If use ftp to transfer files to the system, consider using scp/sftp instead.

1
vote

I implemented port knocking, and have a few probes per day. They don't get a connection, so they go away. I log and report all access to the ports involved.

I have also run fail2ban with Shorewall as a firewall to temporarily blacklist persistent attackers.

If you don't need Internet access to SSH disable it. If you have a few known addresses which need remote access, limit access to those addresses.

Limiting access to authorized-keys can also be helpful.

0
votes

I use pam_abl to temporarily blacklist brute forcers, and it works great. I think it feels better to have authorization in PAM using its own database rather than depend on hosts.deny or iptables.

Another plus is that pam_abl does not depend on scanning log files.

0
votes

It's completely normal these days.
You can set up "burst" limit on firewall for incoming new connections on the SSH port,
or install one of many log parsers a'la fail2ban or change the SSH port ;).

The last one is the easiest one. On heavy loaded machines such break-in attempts can heavy really bad influence to whole system.

--
Regards,
Robert

0
votes

Yes it's normal.

I just changed the ssh port away from the standard 22. My server, my rules :) just edit /etc/ssh/sshd_config, change the port and restart the service. The only down side is that you must remember to add that port to configuration to every ssh client you use.

0
votes
  • Disable root login (In every linux system root user exist so bots can easily guess the user name).After logging in as normal user you can switch to root either by su or sudo.

  • change default port from 22

  • Allow ssh access from known ip's only

  • Use a strong alpha-numeric-password for the user with ssh access

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .