Basically you have both forward and rear-facing VIPs. As an example;
Two Web Servers x.x.x.1 & x.x.x.2 have a client-facing VIP of x.x.x.3, this last one is the one that the DNS records point to and is managed either by the servers themselves or another load-balancer depending on your OS/implementation.
They also have a rear/internal-facing VIP of say y.y.y.1 that load-balances to their internal IPs y.y.y.1 & y.y.y.2. The web servers talk 'down' to the internal tier VIP (z.z.z.3) via these last two interfaces but are spoken 'up' to by the internal servers on this y.y.y.3 VIP.
The internal servers might be z.z.z.1 & z.z.z.2 with an upward/web-facing VIP of z.z.z.3 and although they each talk up to the web servers via the y.y.y.3 VIP and are spoken to by these servers on the z.z.z.3 VIP.
Of course you can then add a fourth/fifth layer down below this to allow your interal servers to talk to DB servers which may have their own VIPs.
Either way you're using load-balancing/VIPs throughout to allow for failure scenarios as you discussed to be dealt with.