0

I have this happening on a few servers with the same firewall rules, so I suspect I am missing something in my iptables configuration but not sure what is wrong. This is happening on some CentOS servers and well as my Ubuntu servers. I've used iptables for years and thought I knew what I was doing... apparently not the case.

I have SSH running on a non-standard port (2022). I have firewall rules to allow access for my personal IPs, followed by blocking specific ports including 2022, then followed by a deny all rule. In the last 3 weeks or so, my logs are showing failed login attempts on SSH from outside IPs that are not in my accept list. I have a VPN service on my laptop so I can try logging in from various countries, IPs, etc and the firewall is not blocking me anymore like it used to. I used my cellphone as a hotspot so I could ensure I had random IP that should be blocked, yet I could still log in to ssh. I used nmap from random IPs and it shows port 2022 as OPEN.

I'm not sure what could be going on, the firewall used to block SSH on not-allowed IP addresses properly and I don't think I've made any changes before this started and I don't have anything like fail2ban complicating things. I've even checked for rootkits as well and nothing shows up. I've Googled a round a lot, but the search results are so noisy with not-quite relevant responses that I've given up and decided to post the question here and hopefully get some better guidance.

This server is running ubuntu 14.04.6 LTS

My allowed IP's are on the 209.xxx and 216.xxx ranges Here are my iptables rules (iptables -L -n):

ACCEPT     all  --  127.0.0.1            0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.1        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.2        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.3        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.4        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.5        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.6        0.0.0.0/0
ACCEPT     all  --  216.xxx.xxx.1        0.0.0.0/0
ACCEPT     all  --  216.xxx.xxx.2        0.0.0.0/0
ACCEPT     all  --  74.xxx.xxx.2         0.0.0.0/0
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp flags:0x3F/0x00
RETURN     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 1/sec burst 2
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp flags:0x3F/0x3F
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:110
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:143
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:465
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:993
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:995
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:2022
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
--------------------------------

Typical Server logs are showing failures like:

Jul 26 05:29:38 SERVERNAME sshd[3536]: Invalid user postgres from 159.89.231.172
Jul 26 05:29:39 SERVERNAME sshd[3536]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=159.89.231.172
Jul 26 05:29:40 SERVERNAME sshd[3534]: Failed password for mysql from 159.89.231.172 port 56352 ssh2
Jul 26 05:29:40 SERVERNAME sshd[3534]: Received disconnect from 159.89.231.172: 11: Normal Shutdown, Thank you for playing [preauth]
Jul 26 05:29:40 SERVERNAME sshd[3538]: reverse mapping checking getaddrinfo for usa1.getlark.com [159.89.231.172] failed - POSSIBLE BREAK-IN ATTEMPT!

Here is the script I created to implement the rules with comments:

          APPEND="sudo /sbin/iptables -A INPUT"
          INSERT="sudo /sbin/iptables -I INPUT"
          OUTPUT="sudo /sbin/iptables -A OUTPUT"


          # drop old rules and start from scratch
          sudo /sbin/iptables -F
          sudo /sbin/iptables -X

          # allow local host
          $INSERT -s 127.0.0.1 -j ACCEPT

          # Allow full access to our approved IPs first:
          $APPEND -s 209.xxx.xxx.1 -j ACCEPT
          $APPEND -s 209.xxx.xxx.2 -j ACCEPT
          $APPEND -s 209.xxx.xxx.3 -j ACCEPT
          $APPEND -s 209.xxx.xxx.4 -j ACCEPT
          $APPEND -s 209.xxx.xxx.5 -j ACCEPT
          $APPEND -s 209.xxx.xxx.6 -j ACCEPT
          $APPEND -s 216.xxx.xxx.1 -j ACCEPT
          $APPEND -s 216.xxx.xxx.2 -j ACCEPT
          $APPEND -s 74.xxx.xxx.2  -j ACCEPT


          # drop Null packets
          $APPEND -p tcp --tcp-flags ALL NONE -j DROP

          # block syn flood attack
                 $APPEND -p tcp --syn -m limit --limit 1/s --limit-burst 2 -j RETURN

          # block recon/Xmas Packets
          $APPEND -p tcp --tcp-flags ALL ALL -j DROP

          # don’t lock me out if I screwed up:
          $APPEND -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

          # Allow/Block our legit services
          $APPEND -p tcp  --dport 80  -j ACCEPT
          $APPEND -p tcp  --dport 110 -j DROP
          $APPEND -p tcp  --dport 143 -j DROP
          $APPEND -p tcp  --dport 443 -j ACCEPT
          $APPEND -p tcp  --dport 465 -j DROP
          $APPEND -p tcp  --dport 993 -j DROP
          $APPEND -p tcp  --dport 995 -j DROP
          $APPEND -p tcp  --dport 2022 -j DROP


          # LAst Rule - Block everything else
          $APPEND -j REJECT --reject-with icmp-host-prohibited
  • Throw this away and use ufw. – Michael Hampton Jul 27 at 19:39
  • Thanks for the idea, but not all my servers can use ufw and iptables is fairly ubiquitous in the *nix world. After using iptables for decades and writing lots of scripts that use iptables, I don't really want to reinvent the wheel. I am just trying to figure out why it suddenly stopped working lately. – Glenn Jul 27 at 22:20
  • In general adding line/ rule numbers to your output helps (that makes discussion easier) but even more useful are packet counters. iptables -L -v -n --line-numbers The counters are incremented whenever the rule is a dispositive match and are very useful to see which rules are triggered and effective or not. Also don’t overlook the fact that if your firewall was broken earlier and allowed everything, the state full rule allowing all established and related connections will continue to allow traffic that would be blocked if the connection would be evaluated as a new connection – HermanB Jul 28 at 9:11
  • Not related to the problem but adding a link about the danger of using REJECT without using INVALID first: doc: document danger of applying REJECT to INVALID CTs – A.B Jul 28 at 20:04
3

Due to my noob reputation I need to ask clarification here.

Can you edit your question to contain also INPUT chain default policy? I ask because of this line:

RETURN   tcp  --  0.0.0.0/0   0.0.0.0/0   tcp flags:0x17/0x02 limit: avg 1/sec burst 2

I'm not an expert, but I have never seen this used in top level. Could be only me though. What is said about RETURN in iptables manpage:

RETURN means stop traversing this chain and resume at the next rule in the previous (calling) chain. If the end of a built-in chain is reached or a rule in a built-in chain with target RETURN is matched, the target specified by the chain policy determines the fate of the packet.

I have a hunch that your problem might have something to do with this functionality.

| improve this answer | |
  • You may be correct. I was getting a lot of syn flooding and found that rule in the internet so I just copied it without actually understanding all of the implications. I have edited my post to show the script used to implement my rules (with comments) – Glenn Jul 27 at 23:28
  • This solved the problem. I commented out: ``` /sbin/iptables -A INPUT -p tcp --syn -m limit --limit 1/s --limit-burst 2 -j RETURN ``` and now the firewall is working properly! Thank you. Moral of the story: Don't implement a firewall rule that you don't completely understand. – Glenn Jul 28 at 21:49
  • Good thing it works! All the best :) – Janne Paalijarvi Jul 28 at 21:57
1

It's hard for me to give an outright answer given the provided data, but my approach would be to:

  1. Trim down iptables down to just two rules: Allow a single IP address range (the one you are connecting from), and deny everything else. Be careful to not lock yourself out of the server! Having console access or a backup plan could be a life saver. If you are confident enough, here's an example on how to proceed. Proceed with caution!:

You can issue commands similar to:

sudo iptables-save > /tmp/original-rules.txt
sudo iptables -F 
sudo iptables -X
sudo iptables -A INPUT -s 209.xxx.xxx.1 -j ACCEPT
sudo iptables -A INPUT -j DROP

The above sequence of commands saves your existing rules, does a basic cleanup of all chains, adds a single IP address or address range to the INPUT chain, and adds a drop "any" rule to the INPUT chain.

Restoring the original rules can be achieved with:

sudo iptables-restore < /tmp/original-rules.txt
  1. Test the two-rule setup:

2a. If it works, you know your problem lies somewhere in the original rules. Slowly keep adding rules and test them, until you are satisfied. You can also use logging to understand which rule is matched.

2b. If your server is still reachable while having only two rules from any other IP address besides the one(s) you allow, you then know it's not a rules issue, but something is up with iptables itself. Perhaps it's not loading correctly, or a previous state of iptables is running in memory.

| improve this answer | |
  • I agree with your methodology for troubleshooting the problem. I have been hesitant because it is a production webserver and I would need to do this at a really off hour so I wouldn't cause a disruption. (and I've been lazy). If no one has an "A-ha! your problem is x" type answer, I will take these steps and likely figure it out. I guess the odd thing is that I wrote and tested these rules on a non-production machine and I remember them working and now for some reason they aren't. It started with a CentOS server (same script) and now it's happening on the Ubuntus. – Glenn Jul 27 at 23:37
0

Original poster accepted my earlier answer, but here is a bit more in-depth:

I diagnosed and debugged a bit. Here is the what I did first:

  1. Installed Ubuntu 14.04 as VMware virtual machine
  2. Modified /etc/ssh/sshd_config to read: Port 2022
  3. Rebooted machine
  4. Marked down VM IP address, which was 192.168.241.171

Next I took your firewall script and removed the xxx entries. I call the modified script "original" script.

I did a couple of reboot / script modifications / script run / Putty connect cycles. Here are scenarios and results:

  • No modifications to "original" script: Putty connection to 192.168.241.171 port 2022 was successful
  • Removed the stanza $APPEND -p tcp --syn -m limit --limit 1/s --limit-burst 2 -j RETURN from "original" script: Putty connection to 192.168.241.171 port 2022 was blocked
  • Added before #allow local host" this: /sbin/iptables -P INPUT DROP` to the "original" script: Putty connection to 192.168.241.171 port 2022 was blocked
  • Added before #allow local host" this: /sbin/iptables -P INPUT ACCEPT` to the "original" script: Putty connection to 192.168.241.171 port 2022 was successful

I think I know what has happened. The user forgot to set the INPUT chain default policy to DROP. Some examples about blocking syn flood using the RETURN statement expect default INPUT chain policy set to DROP and the user had adapted such example without setting the policy. I further speculate that the firewall script has never worked fully and what has been guarding sshd from external users is non-standard port 2022 (but now the bad guys are also knocking on 2022).

PS. If one changes the default policy, it should be done after work hours to negate immediate effects to people, then tested and adapted around new problems if any.

| improve this answer | |
  • You are correct. I grabbed that line from someone's syn flood script and paid no attention to how it worked. I'm putting the pieces together now... it was a few weeks ago that I was having the syn flooding and it wasn't long after that when the ssh authentication failures started happening. I really appreciate your excellent explanation. – Glenn Jul 28 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.