I have 6 nodes that have internet access on eth1 and private access to one another on eth0. Currently I have firewall rules for eth0, for things like memcached and NFS. Is this necessary? It's a real headache as NFS for example communicates on loads of different ports, and I recently introduced glusterfs which needs more still. Is the headache of figuring out what backend ports to unblock worth the security enhancement?
I should mention that I will of course still have a firewall rule on eth0 to block servers owned by others in the same datacenter.