I see two major troubles to deal with.
Having a public subnet, your pfsense box becomes essentially a router. You need to deal with your uplink provider ISP, to route the packets of your subnet to the external IP of your pfsense box. If it somehow won't work (for example, they aren't cooperative or don't even understand what you want), there are some tricks (for example, proxy arp), but hopefully you won't need these.
However, to the direction of OPT1, you want a NAT-ting router. In FreeBSD, the base platform of pfsense, NAT and firewalls are more loosely coupled as usual. You have to count with some unexpected interaction between your NAT and firewall, which are all documented, but mostly on different freebsd sources, and not really in the pfsense documentation. It is important to learn the question, the order in which the NAT and the FW rules to the incoming and outgoing packets are applied.
My opinion is that you don't need multiple subnets for that in an ESXi environment, however you can if you want it on some - here undetailed - reason. It complicates the problem, and the gain is zero to little. The advantages what I can see:
Security, your public and internal networks wouldn't see eachother, thus your public network could work essentially as a DMZ. However, in an ESXi environment, the machines are protected from network eavesdropping by default (ESXi admins can turn it off for per-machine in the VM settings, but induvidual operators can't). On an ESXi subnet, by default you can't see the packets which have nothing to do with you, not even if you are running a packet sniffer on a VM.
You could provide different dhcp services on LAN and OPT1, which might be essential in an automatized virtual hosting environment. However, in this case you can't switch LAN and OPT1 together, furthermore you might have additional challenges to solve the LAN-OPT1-WAN communcation in all direction, securely.
In my opinion, the best is to utilize the defense of the ESXi, to make your pfsense simpler. If you do this, do it so:
- Remove OPT1.
- Set the IP of LAN to 192.168.1.1.
- Configure its dhcp to serve the IPs in the range 192.168.1.2 - 192.168.1.254 to the VMs. (Note, 192.168.1.1 will be you, 192.168.1.255 will be the network broadcast). Last time I've did it, I had troubles to set up a pfsense dhcpd to a different IP range on which it is running, this is why you need to first set it up.
- Set up a virtual IP on LAN to your pfsense public ip (22.214.171.124) and subnet (/29). Easily you can't have a dhcp server for this subnet, this is a disadvantage of this idea.
If you've done everything correctly, your private VMs should see eachother, and also your public VMs should see eachother, but there won't be more routing yet.
- Set up the NAT rules for LAN. Here you have another problem: on the top of the NAT settings, you need to set up your NAT type. Based on the NAT type you've chosen, it will autogenerate some NAT rules for you. As I last time did it, I could not understand, what exactly these rules are doing, and also I could not replicate them with manual settings, but they seemed to be needed for the seamless interaction between your NATted and routed subnets.1
- Set up also the routings in your firewall. Note: in freebsd, and also in pfsense, everything is blocked by default, and anything you want to pass, need to have to find its "pass" rule! Beware, don't close out yourself! (Although in the case of an ESXi pfsense VM, it is easy to fix it with console access.)
Contrary the common router/switch setups, here in the ESXi world you have the major trouble that not having real PROMISC access, you also can't really trace, what is happening to your packets. However, in the pfsense top menu, there is a Diagnostics -> Packet tracer, and there you can really trace everything. Never try to interact blindly with your system, use always the capture data to understand, what is wrong with your current config and where to go further!
Note also, using many ESXi one might develop the custom to like to bridge their subnets together. It is a very different custom than we do in the ip routing configuration, where bridge has its role and is applied where it should do.
You can put the NAT and routing subnets to different subnets, the steps will be roughly the same.
1It is very likely, that here lies the real cause of the problem.