I am operating a server hosting a set of services, each run in a separate Docker container. In addition, there is a KVM running pfSense acting as firewall. The firewall has a physical interface that is connected to the external network and a virtual network card that is connected to the internal container network, using MACVLAN Docker-side, so each container has its own IP address, but all of them are in the same subnet.

For security reasons, the containers need to be isolated and shall not be able to communicate with each other principally (just with the external network). For this, MACVLAN is configured in VEPA mode, which allows traffic from and to the parent device, but not to other addresses on the same parent device.

Now, I would like to allow specific traffic between specific containers, so pfSense must route the traffic to the same interface it received the traffic, considering the configured firewall rules (read, if incoming traffic on the internal interface matches a PASS rule it shall be forwarded to a host on the same interface / same subnet).

I can't seem to get that scenario working (no traffic between the hosts on the internal interface, traffic from and to the external network works as expected). Any ideas on how to proceed from here?

Is there any configuration item in FreeBSD in general or pfSense specifically the prevents such scenarios, like "filter traffic on own interface" or "in practice this should not happend because traffic is forwarded on the switch in front of the router because it is the same subnet so do nothing with it"?

Interestingly enough, pfSense does not even reply to the ARP request (which might have the same reason):

[root@server ~]# ip r
default via dev server proto static metric 410 dev server proto kernel scope link src metric 410

21:52:49.651286 ARP, Request who-has tell, length 28
21:52:50.673895 ARP, Request who-has tell, length 28
21:52:51.697860 ARP, Request who-has tell, length 28
21:52:52.721992 ARP, Request who-has tell, length 28

I would assume a response with the MAC of the interface. The trace was made on the firewall on that interface (PING from the firewall to works as expected).

When adding the entry manually I can see the ICMP echo request, but no reply:

[root@server ~]# arp -s 02:42:0a:00:14:04                ether   02:42:0a:00:14:04   CM                    server

22:00:21.403515 IP > ICMP echo request, id 5622, seq 1, length 64
22:00:22.450162 IP > ICMP echo request, id 5622, seq 2, length 64
22:00:23.473790 IP > ICMP echo request, id 5622, seq 3, length 64
22:00:24.497803 IP > ICMP echo request, id 5622, seq 4, length 64

It depends in part on how you intend for one container to reference another. It would be worth considering whether this network topology is suitable for your use case and security policy.

If you wish for container A to address container B directly by hostname or IP address, this needs to follow Layer 2 switching and Layer 3 routing:

  • If they are in the same subnet then they will only be able to address each other directly through the switch (which will be blocked by VEPA as you suggested). The pfSense host will only see the traffic if:
    • the traffic destination IP is outside of the container subnet, in which case the container will send the traffic to the default gateway to be routed; or
    • the traffic is directly addressed to the pfSense hostname/IP address.
  • If you placed each container on a different subnet (or even separate virtual networks) then traffic between them would be routed via the gateway (pfSense). This would place pfSense in control of firewall policy.
  • If you can determine a policy of which groups of containers are permitted to access each other directly, it would make more sense to group these into virtual/docker networks; then the problem goes away. In this case, you're aligning the network topology with the security policy. This is generally easier to get right, maintain, and for others to reason with.

You could look at Docker Macvlan's 802.1q trunk bridge mode which might make it easier to attach multiple container networks to the same libvirt/pfSense interface.

Alternatively, it may happen that there are only a very narrow and consistent set of ways in which containers access each other. In this case, you could consider port forwarding. You would:

  • identify a specific container service which you wish to be accessible to other containers
  • on pfSense, forward incoming traffic on a particular port of the MACVLAN interface to the IP/port of the desired container
  • when other containers need to access the service, they use the IP address (and nominated port) of the pfSense host (not the target container)

In this case, the other containers don't know anything about where the target service is - it may as well be hosted directly on the pfSense host. Note however that this doesn't scale well, and would only work for simple TCP/UDP traffic (FTP could be painful to set up).

There may well also be some other Docker or libvirt networking features which would allow you to define more detailed firewall policy between containers on a virtual network, though I haven't looked deeply into this myself.

| improve this answer | |
  • Aligning the network topology with the security policy sounds reasonable, the problem I see is that this bypasses the firewall, so there is no fine-grained control over the traffic. 802.1Q on the other hand means that there will be a VLAN for each container, and thus a interface for each container, which requires bridging within the KVM and a duplication of the rules for each container, which increases the maintainence effort and the required processing power (and thus power draw) - the same goes with port forwarding. – Lars Apr 7 '19 at 17:11
  • Do you have any idea whether FreeBSD / pfSense allows for activating hairpinning / reflective relay functionality on a port (resp on a bridge, if this is not possible). To me this seems to be the reason ("... if the destination is on the same segment as the origin segment, the bridge will drop the packet because the receiver has already had a chance to see the frame."). – Lars Apr 7 '19 at 17:11
  • Fair enough, but perhaps that's then your security policy - isolate everything. Absolutely true about the maintenance effort. However I don't think the processing/power concern is mitigated in your original case (esp. for port fowarding), since any topology where pfSense is able to block/allow the traffic will require it to forward packets and apply a ruleset. – Samuel Jaeschke Apr 9 '19 at 0:12
  • Unfortunately I don't have a deep understanding of iptables nuances specific to FreeBSD. I've done "router on a stick", but that instead nvolves different subnets and aliased interfaces. What you're describing almost sounds like SDN, which would be overkill. I do wonder whether there's an existing solution to this problem within the Docker ecosystem though. – Samuel Jaeschke Apr 9 '19 at 0:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.