Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I've been trying to secure my virtual machines on my esx server by putting them behind a transparent bridge with 2 interfaces, one in front, one at the back.

My intention is to put all the firewall rules in one place (instead of on each virtual server).

I've been using as bridge a blank new virtual machine based on arch linux (but I suspect it doesn't matter which brand of linux it is).

What I have is 2 virtual switchs (thus two Virtual Network, VN_front and VN_back), each with 2 types of ports (switched/separated or promiscious/where the machine can see all packets).

On my bridge machine, I've set up 2 virtual NIC, one on VN_front, one on VN_back, both in promisc mode.

I've created a bridge br0 with both NIC in it:

brctl addbr br0
brctl stp br0 off
brctl addif br0 front_if
brctl addif br0 back_if

Then brought them up:

ifconfig front_if 0.0.0.0 promisc
ifconfig back_if 0.0.0.0 promisc
ifconfig br0 0.0.0.0

(I use promisc mode, because I'm not sure I can do without, thinking that maybe the packets don't reach the NICs)

Then I took one of my virtual server sitting on VN_front, and plugged it to VN_back instead (that's the nifty use case I'm thinking about, being able to move my servers around just by changing the VN they are plugged into, without changing anything in the configuration).

Then I looked into the macs "seen" by my addressless bridge using brctl showmacs br0 and it did show my server from both sides:

I get something that looks like this :

port no mac addr                is local?       ageing timer
  2     00:0c:29:e1:54:75       no                 9.27
  1     00:0c:29:fd:86:0c       no                 9.27
  2     00:50:56:90:05:86       no                73.38
  1     00:50:56:90:05:88       no                 0.10
  2     00:50:56:90:05:8b       yes                0.00  << FRONT VN
  1     00:50:56:90:05:8c       yes                0.00  << BACK  VN
  2     00:50:56:90:19:18       no                13.55
  2     00:50:56:90:3c:cf       no                13.57

the thing is that the server that are plugged in front/back are not shown on the correct port.

I suspect some horrible thing happening in the ARP-world... :-/

If I ping from a front virtual server to a back virtual server, I can only see the back machine if that back machine pings something in the front. As soon as I stop the ping from the back machine, the ping from the front machine stops getting through...

I've noticed that if the back machine pings, then its port on the bridge is the correct one...

I've tried to play with the arp_ switch of /proc/sys, but with no clear effect on the end result... /proc/sys/net/ipv4/ip_forward doesn't seem to be of any use when using a bridge (seems it's all taken care of by brctl) /proc/sys/net/ipv4/conf//arp_ don't seem to change much either... (tried arp_announce to 2 or 8 - like suggested elsewhere - and arp_ignore to 0 or 1 )

All the examples I've seen have a different subnet on either side like 10.0.1.0/24 and 10.0.2.0/24... In my case I want 10.0.1.0/24 on both side (just like a transparent switch - except it's a hidden fw ).

Turning stp on/off doesn't seem to have any impact on my issue.

It's as if the arp packets where getting through the bridge, corrupting the other side with false data... I've tried to use the -arp on each interface, br0, front, back... it breaks the thing altogether... I suspect it has something to do with both side being on the same subnet...

I've thought about putting all my machine behind the fw, so as to have all the same subnet at the back... but I'm stuck with my provider's gateway standing at the front with part of my subnet (in fact 3 appliance to route the whole subnet), so I'll always have ips from the same subnet on both side, whatever I do... (I'm using fixed front IPs on my delegated subnet).

I'm at a loss... -_-'' Thx for your help.

(As anyone tried something like this? from within ESXi?)

(It's not just a stunt, the idea is to have something like fail2ban running on some servers, sending their banned IP to the bridge/fw so that it too could ban them - saving all the other servers from that same attacker in one go, allowing for some honeypot that would trigger the fw from any kind of suitable response, and stuffs of the sort...

I am aware I could use something like snort, but it addresses some completely different kind of problems, in a completely different way... )

share|improve this question
    
I've tried to build a complete new setup from scratch: in that new setup, my hiddenfw/bridge is still connected to the "big internet", it still has the gw in the same subnet as the servers, but at the back, I've set up a completely new server, with only one NIC so as to be sure it doesn't contaminate the "front" with packets routing I don't know how (I've also disabled ipv6 altogether on the fw)... It still does the same as before, with the mac address of the back server appearing on the front port of the bridge. The MAC goes to the back if I ping the server from the fw, tho... -_- and back.. –  Mink Jun 1 '14 at 13:10
    
I'm beginning to thing I have a TTL problem... Somewhere on the network, there must be some kind of switch/router (which belong to the hosting company) who keeps remembering some old arp neigbours, and poisons my network with junk data... (will trace of it with tcpdump -nnetqi ). (This hunch comes from the fact that going away for an hour or so to relax seem to have make the ping from home to the datacenter finally go through)... Will keep you posted. –  Mink Jun 1 '14 at 14:51
    
Great, I broke something... created a loop somehow, had my whole network disabled by the hosting company that give me the uplink to the internet... Had to go to the datacenter, gain access through a broken network - the uplink was down, the lan was a mess couldn't connect to anything, killed the virtual machine that was causing the loop, had the noc restart my link (Should this happen to you, what I did to finally gain access, was to unplug the physical machine from the network, plugged the maintenance PC directly into the server physical NIC, thus avoiding any problems with the loop) –  Mink Jun 1 '14 at 20:46

1 Answer 1

If I understand correctly, your desired network is:

         +----------+  +-----+  +---------+
<--inet--+ VN_front +--+ br0 +--+ VN_back |
         +--+-------+  +-----+  +------+--+
            |                          |       
            |                          |       
         +--+----------+    +----------+--+
         | VM          |    | VM          |
         | 10.0.1.2/24 |    | 10.0.1.3/24 |
         +-------------+    +-------------+

With internet gateway via the arrow to the left?

And you use ebtables or iptables on the bridge to firewall what's in the "back" part of the network?

I'm sure this could be made to work, but I wouldn't do it this way. It's a complex and uncommon setup. Like programmers say, there are always two people who will debug anything you create - you, and you in six months when you've forgotten it all.

All the examples you've seen with two subnets (and presumably much simpler firewalling) are the common practice for a reason - they work first time, they are obvious to configure, to change, to maintain, and to troubleshoot.

Simplicity is everything in networking. Avoid building a complex network. Be proud of your simple and stable network which a well-trained monkey (or even an MCSE) could run. Get your boss to give you KPIs on the number of tickets you don't generate to prove and give incentives for that stability.

(If your MACs are appearing as being behind the wrong bridgeport, it'll be because you have created a network loop and ARP requests are being broadcast to both interfaces of the bridge, and the last interface received learns where the MAC is. From your later comments, it sounds like you have done just that.)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.