I am planning to deploy an infrastructure of 11 nodes using Opscode Chef, providing for a high availability web application. I would like to spread the nodes across datacentres for availability, for which I'm thinking rrDNS which resolves to one of two load-balancers, each in a separate datacentre with its own clustered nodes (providing the application with nginx, memcached and Sphinx). A third DC will be used for a MySQL master/slave arrangement, as I have read replication does not perform well over WAN. This solution enables no single point of failure.
My question pertains to how these nodes should connect to one another? All the information transferred by these services is generally expected to go over short LAN connections, hence no in-built security is provided, meaning I will need to provide this on the links themselves.
I was thinking of doing this with SOCKS tunnelling or VPN. The latter would double as increasing security for the nodes themselves, as then they do not need to expose several services on their Internet interface, instead just exposing say an instance of OpenVPN.
What are the thoughts on solutions for providing links between nodes in this sort of infrastructure?