There are advantages when you can deploy one server A so that it serves as the IP gateway for machines B, C, and D. In a real machine room, for example, A can balance incoming customer connections by forward packets to B, C, and D without even pausing to rewrite them, so long as B, C, and D are all programmed to think of the service's public IP address as their own — and all reply packets can flow directly back to customers without returning through the load balancer.
Less ambitiously, something like HAProxy can be used with the Linux Tproxy kernel extension on the gateway machine A, so that incoming load-balanced connections can have their packets forwarded to B, C, and D without changing the source IP address, but only the destination IP address. While this requires return packets to be rewritten, and B, C, and D have to be configured to use A as their IP gateway, it solves the problem of how to load-balance without losing the ability to detect and log customer IP addresses on servers B, C, and D.
My question, then:
Which cloud hosting providers allow machines to be placed on a virtual LAN such that real link-level freedom is provided to send packets between machines simply based on MAC address, and without regard for IP addresses?
If machine B is going to use machine A as an IP gateway, then obviously machine A will be receiving lots of packets from machine B that do not have A's IP address, but the address of other hosts out on the Internet with which B wishes to communicate.
I spent an hour last night experimenting with the Virtual Private Cloud (VPC) service offered by Amazon, but found that having machine A serve as a gateway was not possible — although I could tell B that A was its IP gateway, and turn on
ip_forward on A, the packets that B tried to send off-LAN never made it to A, apparently because Amazon requires both the MAC address and the IP address to be correct before passing a packet from B to A.
Which (if any) cloud hosting providers would let me set up a LAN so that machines can pass packets purely based on MAC addressing?