1

I'm investigating some anomalies I found on a server I manage after installing Snort IDS.

I have ~10 Ubuntu production webservers that are set up with Ansible and regularly provisioned. One of them is displaying very strange routing of requests.

Lets say the server's IP public is 111.1.1.11.

  • It's a Digital Ocean droplet.
  • It only has one NIC.
  • ifconfig shows the NIC with public IP 111.1.1.11.

When I look at the logs in Snort, and also when I monitor the network traffic with iftop I see that:

  • All of the traffic I expect (ssh, https, an SQL port) is being routed to a host at a private address in the 10.0.0.0-10.255.255.255 range. Let's say it's: 10.20.0.10.
  • Traffic on other ports is not being redirected there, and is resolving to ports on 111.1.1.11.

If I look through my iptables rules there are no mentions of this host 10.20.0.10, or anything unusual with routing. netstat is not showing anything unusual. If I ping 10.20.0.10 I can see there is something there. I can't see any processes running on the server that are unexpected.

I can't figure out why or how traffic from external addresses is being routed to this private address. I've checked the other 10 servers (which are all configured exactly the same, via Ansible) and they don't have this strange routing behaviour. All their traffic resolves to regular ports on their respective public IPs.

Any ideas?

Update:

It seems like the network interface configuration is non-standard on our servers that are hosted with Digital Ocean.

Standard config:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

DO config:

auto lo
iface lo inet loopback
    dns-nameservers 67.207.67.2 67.207.67.3 # This is custom Digital Ocean DNS

auto eth0
iface eth0 inet static
    address 111.1.1.11/20
    gateway 111.1.0.1

# control-alias eth0
iface eth0 inet static
    address 10.20.0.10/16

So that's part of the mystery solved. Our other DO hosted servers have a similar config (with different IPs, obviously).

The remaining issue is that our other DO-hosted servers don't route any incoming traffic to their addresses listed under "control-alias eth0", so I'm still confused as to why one server does, and the others don't.

  • Please check the output of iptables-save. By default the iptables command shows only the filter table. Run the tcpdump and check the traffic. – Anton Danilov Aug 4 '19 at 10:10
  • @AntonDanilov yes, I used iptables -S which I think gives the same output as iptables-save. Either way, there's nothing there that mentions the phantom host. – bugged Aug 4 '19 at 10:17
0

Turns out that private address is what Digital Ocean calls an "Anchor IP", used internally for routing traffic from Floating IPs.

Somebody else had added a floating IP to that Droplet, and I didn't realise.

/facepalm

Well at least I've learnt something new, and maybe this will help some other confused syadmin some day.

Possible solutions:

  • Get rid of the Floating IP
  • Adapt Snort configuration (it was throwing a lot of false-positives for that traffic, and would have blocked most of it)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.