I am operating a server hosting a set of services, each run in a separate Docker container. In addition, there is a KVM running pfSense acting as firewall. The firewall has a physical interface that is connected to the external network and a virtual network card that is connected to the internal container network, using MACVLAN Docker-side, so each container has its own IP address, but all of them are in the same subnet.
For security reasons, the containers need to be isolated and shall not be able to communicate with each other principally (just with the external network). For this, MACVLAN is configured in VEPA mode, which allows traffic from and to the parent device, but not to other addresses on the same parent device.
Now, I would like to allow specific traffic between specific containers, so pfSense must route the traffic to the same interface it received the traffic, considering the configured firewall rules (read, if incoming traffic on the internal interface matches a PASS rule it shall be forwarded to a host on the same interface / same subnet).
I can't seem to get that scenario working (no traffic between the hosts on the internal interface, traffic from and to the external network works as expected). Any ideas on how to proceed from here?
Is there any configuration item in FreeBSD in general or pfSense specifically the prevents such scenarios, like "filter traffic on own interface" or "in practice this should not happend because traffic is forwarded on the switch in front of the router because it is the same subnet so do nothing with it"?
Interestingly enough, pfSense does not even reply to the ARP request (which might have the same reason):
[root@server ~]# ip r default via 10.0.20.1 dev server proto static metric 410 10.0.20.0/24 dev server proto kernel scope link src 10.0.20.2 metric 410 21:52:49.651286 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28 21:52:50.673895 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28 21:52:51.697860 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28 21:52:52.721992 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28
I would assume a response with the MAC of the 10.0.20.0/24 interface. The trace was made on the firewall on that interface (PING from the firewall to 10.0.20.4 works as expected).
When adding the entry manually I can see the ICMP echo request, but no reply:
[root@server ~]# arp -s 10.0.20.4 02:42:0a:00:14:04 10.0.20.4 ether 02:42:0a:00:14:04 CM server 22:00:21.403515 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 1, length 64 22:00:22.450162 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 2, length 64 22:00:23.473790 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 3, length 64 22:00:24.497803 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 4, length 64