I am trying to implement the following in a lab subscription, as described in https://learn.microsoft.com/en-us/azure/architecture/guide/networking/ipv4-exhaustion#implement-snat-via-third-party-nvas (Method 1: Nonroutable landing zone spoke virtual networks / Outbound dependencies / Implement SNAT via third-party NVAs):
At this stage, it is purely a learning exercise for me, but the eventual goal is to support interconnection with a series of existing on-premises estates, where VPNs to each will be established either to a VPN Gateway or (more likely) to a third-party NVA in the VNET designated Routable Hub.
In the eventual setup there will be many Landing zones and, for autonomy (read: political) reasons, I will not have oversight of the IP address spaces used by each of the LZ spokes, so am looking to corral the outbound traffic from those spokes into an address space I can manage, using a per-zone Routable LZ hub VNET (which I can control) as an intermediary.
In the lab, I have successfully established:
- VPN between On-premises and Routable hub (A), using a third-party NVA
- VNET peering between Routable hub (A) and Routable LZ hub (B)
- a routing table exists in (B) to route the on-premises subnet via the NVA
- a VM in (B) can ping the NVA in (A)
- a VM in (B) can ping an on-premises device, routing via the NVA
- VNET peering between Routable LZ hub (B) and LZ spoke (C)
- a VM in (B) can ping a VM in (C), via the default route
- a VM in (C) can ping a VM in (B), via the default route
I am now trying to apply the principles described in Outbound dependencies, where a VM in (C) might need access to an on-premises device. As such, I have added:
- an NVA device to (B), specifically a one-armed pfSense VM from the Azure Marketplace
- a routing table to (C), which routes the on-premises subnet via the pfSense
- I have configured an Outbound NAT rule on the pfSense to translate anything from (C)'s address space to the pfSense's address in (B)'s address space.
With this in place, I can see:
- traffic from the VM in (C) arrive at the pfSense
- the pfSense apply SNAT to its address, and dispatch to (B)'s default gateway
- the NVA in (A) accepts the traffic from (B) with the source address of the pfSense
- the NVA send across the VPN, targeting the on-premises device
- the target on-premises device sees the traffic, accepts it and replies
- the NVA in (A) receives the reply traffic and forwards to the pfSense in (B), which the pfSense accepts
- the pfSense in (B) reverses the SNAT and forwards to the address of the VM in (C)
But the reply traffic never appears on the VM in (C).
Below is some of the "evidence" of what I have said I can see:
- The VM IP is 10.0.16.254; it exists in (C), subnet 10.0.16.0/20
- The pfSense IP is 172.23.2.6; it exists in (B), subnet 172.23.2.0/24
- The target on-premises device IP is 172.18.4.80; subnet 172.18.0.0/16 (via VPN)
If I am reading it correctly, the pfSense correctly SNATs and un-SNATs the traffic, but the traffic is dropped somewhere during the reply, between the pfSense in (B) and the VM in (C).
There are NSGs on the interfaces of the pfSense and the VM, but I believe these are correct -- insofar as they both have a rule which allows traffic from 172.18.0.0/16 to VirtualNetwork at some point before the three default rules.
(I recognise that, within NSGs, VirtualNetwork does not encompass include the on-premises subnet, as it is not associated directly with any VNET, but I believe should include all peered VNET subnets.)
Other things that may be of note (but may be red-herrings):
- The VM in (C) is configured to log all accepted and blocked packets
- A tcpping from the pfSense in (B) to a valid service on the VM in (C) seem to reach the VM and be logged
- A tcpping from pfSense to an invalid service on the VM seem not to be logged (but it may be a delay in log flush)
I'm sure I've missed something simple but what else might be preventing the reply traffic reaching the VM in (C), after it has been un-SNAT'd by the pfSense in (B)...?
PS. Note that, since this is as much a learning exercise for me as anything else, I am particularly interested in the model as diagrammed by Microsoft at this point, rather than any alternatives -- including alt diagrams on that same page
PPS. Also note that, while the target diagram indicates two SNAT NVAs with a Front-end Load Balancer IP, I currently only have one SNAT NVA and no LB IP -- I am just trying to get the principles working initially and will look at high-availablity later. If the LB IP is providing the missing piece of the puzzle, then please advise.