0

I have set up my Debian server on HP ProLiant 360 G4 yesterday. I started off with minimum services, running only SSH and Apache, all on standard ports with default configurations.

About an hour ago, I noticed odd behaviour of the system. The latency raised noticeably and I could not perform remote reboot. I managed to get the server off network in about 15 minutes.

I have been going through the logs and found these entries in auth.log:

Apr  3 17:31:35 karel sshd[25941]: input_userauth_request: invalid user takeuchi [preauth]
Apr  3 17:31:35 karel sshd[25941]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:31:35 karel sshd[25941]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:31:37 karel sshd[25941]: Failed password for invalid user takeuchi from 70.61.237.202 port 53004 ssh2
Apr  3 17:31:37 karel sshd[25941]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:31:39 karel sshd[25943]: Invalid user takeuchi from 70.61.237.202
Apr  3 17:31:39 karel sshd[25943]: input_userauth_request: invalid user takeuchi [preauth]
Apr  3 17:31:39 karel sshd[25943]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:31:39 karel sshd[25943]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:31:41 karel sshd[25943]: Failed password for invalid user takeuchi from 70.61.237.202 port 30756 ssh2
Apr  3 17:31:41 karel sshd[25943]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:31:42 karel sshd[25945]: Invalid user takeuchi from 70.61.237.202
Apr  3 17:31:42 karel sshd[25945]: input_userauth_request: invalid user takeuchi [preauth]
Apr  3 17:31:42 karel sshd[25945]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:31:42 karel sshd[25945]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:31:45 karel sshd[25945]: Failed password for invalid user takeuchi from 70.61.237.202 port 43388 ssh2
Apr  3 17:31:45 karel sshd[25945]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:31:46 karel sshd[25947]: Invalid user takeuchi from 70.61.237.202
Apr  3 17:31:46 karel sshd[25947]: input_userauth_request: invalid user takeuchi [preauth]
Apr  3 17:31:46 karel sshd[25947]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:31:46 karel sshd[25947]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:31:49 karel sshd[25947]: Failed password for invalid user takeuchi from 70.61.237.202 port 29640 ssh2
Apr  3 17:31:49 karel sshd[25947]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:31:50 karel sshd[25949]: Invalid user takeuchi from 70.61.237.202
Apr  3 17:31:50 karel sshd[25949]: input_userauth_request: invalid user takeuchi [preauth]
Apr  3 17:31:50 karel sshd[25949]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:31:50 karel sshd[25949]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:31:52 karel sshd[25949]: Failed password for invalid user takeuchi from 70.61.237.202 port 56323 ssh2
Apr  3 17:31:52 karel sshd[25949]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:31:54 karel sshd[25951]: Invalid user takeuchi from 70.61.237.202
Apr  3 17:31:54 karel sshd[25951]: input_userauth_request: invalid user takeuchi [preauth]
Apr  3 17:31:54 karel sshd[25951]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:31:54 karel sshd[25951]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:31:56 karel sshd[25951]: Failed password for invalid user takeuchi from 70.61.237.202 port 54603 ssh2
Apr  3 17:31:56 karel sshd[25951]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:31:57 karel sshd[25953]: Invalid user takeuchi from 70.61.237.202
Apr  3 17:31:57 karel sshd[25953]: input_userauth_request: invalid user takeuchi [preauth]
Apr  3 17:31:57 karel sshd[25953]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:31:57 karel sshd[25953]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:31:59 karel sshd[25953]: Failed password for invalid user takeuchi from 70.61.237.202 port 30332 ssh2
Apr  3 17:31:59 karel sshd[25953]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:32:01 karel sshd[25955]: Invalid user takeuchi from 70.61.237.202
Apr  3 17:32:01 karel sshd[25955]: input_userauth_request: invalid user takeuchi [preauth]
Apr  3 17:32:01 karel sshd[25955]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:32:01 karel sshd[25955]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:32:03 karel sshd[25955]: Failed password for invalid user takeuchi from 70.61.237.202 port 30855 ssh2
Apr  3 17:32:03 karel sshd[25955]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:32:04 karel sshd[25957]: Invalid user takeuchi from 70.61.237.202
Apr  3 17:32:04 karel sshd[25957]: input_userauth_request: invalid user takeuchi [preauth]
Apr  3 17:32:04 karel sshd[25957]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:32:04 karel sshd[25957]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:32:07 karel sshd[25957]: Failed password for invalid user takeuchi from 70.61.237.202 port 31154 ssh2
Apr  3 17:32:07 karel sshd[25957]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:32:08 karel sshd[25959]: Invalid user wut from 70.61.237.202
Apr  3 17:32:08 karel sshd[25959]: input_userauth_request: invalid user wut [preauth]
Apr  3 17:32:08 karel sshd[25959]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:32:08 karel sshd[25959]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:32:11 karel sshd[25959]: Failed password for invalid user wut from 70.61.237.202 port 59904 ssh2
Apr  3 17:32:11 karel sshd[25959]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:32:12 karel sshd[25961]: Invalid user wut from 70.61.237.202
Apr  3 17:32:12 karel sshd[25961]: input_userauth_request: invalid user wut [preauth]
Apr  3 17:32:12 karel sshd[25961]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:32:12 karel sshd[25961]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:32:14 karel sshd[25961]: Failed password for invalid user wut from 70.61.237.202 port 45945 ssh2
Apr  3 17:32:14 karel sshd[25961]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:32:15 karel sshd[25963]: Invalid user wut from 70.61.237.202
Apr  3 17:32:15 karel sshd[25963]: input_userauth_request: invalid user wut [preauth]
Apr  3 17:32:15 karel sshd[25963]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:32:15 karel sshd[25963]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:32:18 karel sshd[25963]: Failed password for invalid user wut from 70.61.237.202 port 52652 ssh2
Apr  3 17:32:18 karel sshd[25963]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:32:19 karel sshd[25965]: Invalid user wut from 70.61.237.202
Apr  3 17:32:19 karel sshd[25965]: input_userauth_request: invalid user wut [preauth]
Apr  3 17:32:19 karel sshd[25965]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:32:19 karel sshd[25965]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:32:21 karel sshd[25965]: Failed password for invalid user wut from 70.61.237.202 port 34513 ssh2
Apr  3 17:32:21 karel sshd[25965]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]
Apr  3 17:32:23 karel sshd[25967]: Invalid user wut from 70.61.237.202
Apr  3 17:32:23 karel sshd[25967]: input_userauth_request: invalid user wut [preauth]
Apr  3 17:32:23 karel sshd[25967]: pam_unix(sshd:auth): check pass; user unknown
Apr  3 17:32:23 karel sshd[25967]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rrcs-70-61-237-202.central.biz.rr.com 
Apr  3 17:32:24 karel sshd[25967]: Failed password for invalid user wut from 70.61.237.202 port 32538 ssh2
Apr  3 17:32:24 karel sshd[25967]: Received disconnect from 70.61.237.202: 11: Bye Bye [preauth]

Should I be worried about potential system breach? I use strong password for my account and root login was disabled at the time. Will it help if I open different port for SSH?

Thanks!

  • 6
    three recommendations, change SSH service port, setup fail2ban to ban failed SSH authorizations and disable password authentication, use only certificates (public-key auth) – Marek Sebera Apr 3 '14 at 19:53
  • 4
    untargeted "random" ssh attacks by bots are unfortunately very common. Check the log for possible successful logins (or also last -a). More secure is to use ssh key instead of password-based authentication. Changing the port does help although some sysadmins don't like it and discussions about it tend to escalate in a "vim vs emacs" fashion. – LinuxDevOps Apr 3 '14 at 19:56
7

Yes, you're under attack, and yes you should be concerned about a system breach, even with a strong SSH password.

You should:

  1. Implement fail2ban

  2. Move SSH to a non-standard port.

    • Moving SSH won't prevent a dedicated attacker from finding where your service is anyway, but it will foil the bots that are used for the vast majority of these remote brute-force attacks. They only ever go after the default port.

  3. Set up certificate authentication for SSH logins.
  • 1
    Thank you for quick answer. I am GNU/Linux beginner and this is my first server. I thought it would take longer before they start taking me down :(. I have changed the port immediately and switched to key-based authentication which I had prepared. – David Apr 3 '14 at 20:02
  • Just wondering, do they simply try random IPs? I have been online for few hours for the first time with this address. – David Apr 3 '14 at 20:06
  • @David Essentially, yeah, random IPs. It's an indication of just how many of these bots are out there that within a few minutes of bringing a server online (at a publicly accessible address), someone will being trying to connect to the well known ports on your server. SSH, SQL, even RDP these days. – HopelessN00b Apr 3 '14 at 20:09
2

To expand upon HopelessN00b's answer above:

  • You can also throttle the number of attempts by using iptables' limit module. Or, if you're using Shorewall, use its LIMIT action to accomplish the same thing. Between fail2ban and this, my SSH service on the standard port sees little more than the occasional half-hearted guess at a user or two before the would-be attacker just moves on.
  • Use the AllowGroups option in sshd_config to restrict which users are even allowed in in the first place (I create a special group called sshusers for exactly this purpose). Make sure common usernames (especially root, but also other common ones like webmaster, postmaster, etc.) are not in this group.
  • In the same file, ensure you set PermitRootLogin to no. Easily 50% of attempts against my servers try to access root, and this ensure no matter how hard they try those will never succeed. You should always be logging in with your user and using sudo when necessary anyway, so this shouldn't hamper your user of your server.

It's exceedingly unlikely that you yourself are being targeted; most likely it's just a random bot scanning IP ranges that's found your server and is trying to get in. This happens all the time, and it's very common. Just make sure you secure your server, watch for signs of unauthorized access (a monitoring system such as Zabbix that can alert you if e.g. /etc/passwd is changed is an excellent first step), and otherwise don't worry about it.

For an advanced measure, you can employ a technique called "Port Knocking" to completely mask the existence of an SSH service altogether unless someone knows the "secret knock" to reveal it. It's not a simple technique to set up, though, and even less so to use, since there's no SSH client out there that can work with it and thus you have to employ additional tools for that purpose. For that reason I don't really recommend it.

  • I also blacklist entire DROP lists such as that provided by Spamhaus. You have to be careful with this route, though, as these lists are advisory only and you could be cutting off legitimate users (but you're unlikely to do so). spamhaus.org/drop – Kromey Apr 3 '14 at 20:21
1

I would not be concerned about those lines in your log. All they show is somebody trying to guess passwords of some users, that don't exist in the first place.

I don't think those log lines can explain the increased latency. The reason for increased latency can be difficult to find even if you try to pin it down while it is happening. Identifying the reason for an increased latency after the fact may well prove to be impossible. It could have been temporary congestion on a router somewhere between you and the server.

Not being able to reboot the machine is worrying - unless you mean that you simply couldn't establish a connection to the server due to a networking issue.

You can look through other logs for hints about what may have been happening. If you didn't already do so, you should look at /var/log/syslog.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.