2

I have a public IP and and a public /29 subnet assigned to the same NIC in a server running ESXi. I have installed pfSense, as a virtual machine on this ESXi.

I gave three virtual NICs to the pfSense:

  • One is attached to the vSwitch connected with the physical NIC of the ESXi box.
  • The other is bridged to the LAN, which is another vSwitch without a physical NIC connection inside the ESXi box.
  • The third, call it OPT1, is bridged to another LAN, which is another vSwitch without a physical NIC connection.

I assigned the public IP to the WAN IP of the pfSense, and the first IP of my /29 subnet to the OPT1.

I am going to configure pgsense to work as a router for the (virtual) machines on OPT1, and as a NAT-ing firewall for the (virtual) machines on LAN.

I realized that the OPT1 subnet is a public IP subnet, which will need routing, so started reading the available documentation. What I want to achieve is really this:

https://docs.netgate.com/pfsense/en/latest/book/firewall/methods-of-using-additional-public-ip-addresses.html

I started with the OPT1 part, LAN part to be configured later. I tried to follow this howto, and I failed. I am afraid it might be down to how I configured Interfaces/Bridges/OPT1 : Member Interfcaces=WAN,LAN

The symptom is that the VM on the subnet can not connect to or ping pfSense anymore. "Network device is connected, no network access".

Is my OPT1 misconfigured? Some help or a troubleshooting guide would be most appreciated.

  • 1
    Welcome on the ServerFault! As I can see, an important part of your question is that you want the pfsense to work as both a router (for OPT1) and as a NAT-ing firewall (for LAN). After I've understood it, I refactored the first half of your question, and I've also clarified some terminology as they are used in the ESXi/pfsense world. If I suspected that badly, or did any bad, feel free to rollback or improve my edit (you can get the edit history by clicking the "edited .... ago" link beside my name, there you can roll back to any previous version of your post). – peterh Feb 21 at 12:49
2
+50

I see two major troubles to deal with.

  1. Having a public subnet, your pfsense box becomes essentially a router. You need to deal with your uplink provider ISP, to route the packets of your subnet to the external IP of your pfsense box. If it somehow won't work (for example, they aren't cooperative or don't even understand what you want), there are some tricks (for example, proxy arp), but hopefully you won't need these.

  2. However, to the direction of OPT1, you want a NAT-ting router. In FreeBSD, the base platform of pfsense, NAT and firewalls are more loosely coupled as usual. You have to count with some unexpected interaction between your NAT and firewall, which are all documented, but mostly on different freebsd sources, and not really in the pfsense documentation. It is important to learn the question, the order in which the NAT and the FW rules to the incoming and outgoing packets are applied.

My opinion is that you don't need multiple subnets for that in an ESXi environment, however you can if you want it on some - here undetailed - reason. It complicates the problem, and the gain is zero to little. The advantages what I can see:

  1. Security, your public and internal networks wouldn't see eachother, thus your public network could work essentially as a DMZ. However, in an ESXi environment, the machines are protected from network eavesdropping by default (ESXi admins can turn it off for per-machine in the VM settings, but induvidual operators can't). On an ESXi subnet, by default you can't see the packets which have nothing to do with you, not even if you are running a packet sniffer on a VM.

  2. You could provide different dhcp services on LAN and OPT1, which might be essential in an automatized virtual hosting environment. However, in this case you can't switch LAN and OPT1 together, furthermore you might have additional challenges to solve the LAN-OPT1-WAN communcation in all direction, securely.

In my opinion, the best is to utilize the defense of the ESXi, to make your pfsense simpler. If you do this, do it so:

  1. Remove OPT1.
  2. Set the IP of LAN to 192.168.1.1.
  3. Configure its dhcp to serve the IPs in the range 192.168.1.2 - 192.168.1.254 to the VMs. (Note, 192.168.1.1 will be you, 192.168.1.255 will be the network broadcast). Last time I've did it, I had troubles to set up a pfsense dhcpd to a different IP range on which it is running, this is why you need to first set it up.
  4. Set up a virtual IP on LAN to your pfsense public ip (192.0.3.1) and subnet (/29). Easily you can't have a dhcp server for this subnet, this is a disadvantage of this idea.

If you've done everything correctly, your private VMs should see eachother, and also your public VMs should see eachother, but there won't be more routing yet.

  1. Set up the NAT rules for LAN. Here you have another problem: on the top of the NAT settings, you need to set up your NAT type. Based on the NAT type you've chosen, it will autogenerate some NAT rules for you. As I last time did it, I could not understand, what exactly these rules are doing, and also I could not replicate them with manual settings, but they seemed to be needed for the seamless interaction between your NATted and routed subnets.1
  2. Set up also the routings in your firewall. Note: in freebsd, and also in pfsense, everything is blocked by default, and anything you want to pass, need to have to find its "pass" rule! Beware, don't close out yourself! (Although in the case of an ESXi pfsense VM, it is easy to fix it with console access.)

Contrary the common router/switch setups, here in the ESXi world you have the major trouble that not having real PROMISC access, you also can't really trace, what is happening to your packets. However, in the pfsense top menu, there is a Diagnostics -> Packet tracer, and there you can really trace everything. Never try to interact blindly with your system, use always the capture data to understand, what is wrong with your current config and where to go further!

Note also, using many ESXi one might develop the custom to like to bridge their subnets together. It is a very different custom than we do in the ip routing configuration, where bridge has its role and is applied where it should do.

You can put the NAT and routing subnets to different subnets, the steps will be roughly the same.

1It is very likely, that here lies the real cause of the problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.