The actual deployment of this will happen with a larger number of servers, but for purposes of simplicity, let's assume the following is my future setup (I need +4 reputation to embed images, so here's a link to a diagram: http://stevejenkins.com/downloads/netdiagram.png).
- Using CentOS 5.6
- Each server has 2 NICs (eth0 and eth1)
- All IPs are static (no DHCP issues involved)
We have two sets of servers on two different public subnets (our ISP provides the default gateways for each).
- Servers 1 & 2 have their eth0 connected to Switch A on Public Subnet A using Default Gateway 111.222.333.1.
- Servers 3 & 4 have their eth0 connected to Switch B on Public Subnet B using Default Gateway 444.555.666.1.
Because our ISP charges us for all traffic that goes across subnets, we'd like the servers to be able to use their second NICs to send internal traffic (backups, database queries, etc.) between themselves using statically-assigned local IP addresses (192.168.1.x) on a third switch (Switch C). That way, Server 1 can store backups on Server 4 by sending traffic across Switch C, without incurring any charges.
There's no router or default gateway for the internal network, and from what I've Googled so far, I believe I need to set up static routes on each of the servers for eth1. But I've never had to mess with routing before, and I haven't been able to find anything that explains how to set something like this up (I'm sure it exists and I probably just don't know which keywords to use).
If I can't figure it out, I suppose a lame Option 2 would be to just swap Switch C for a cheap router, which could act as a default gateway at 192.168.1.1. But I'd like to give the routes a shot and learn something new in the process.
So - how do I tell each server to send traffic for 192.168.1.x over eth1 without a default gateway in place?
Thanks in advance!