I have 2 servers under Windows 2008 R2. Each server has 2 network adapters: one - 192.168.1.x network with a gateway, the other one is 10.10.10.x with no gateway.
Server1: Eth1 - 192.168.1.114, netmask 255.255.255.0, gw 192.168.1.200 Eth2 - 10.10.10.1, netmask 255.255.255.0
Server2: Eth1 - 192.168.1.116, netmask 255.255.255.0, gw 192.168.1.200 Eth2 - 10.10.10.2, netmask 255.255.255.0
Internet traffic goes through Eth1 on both servers (the servers are connected to a switch), redundancy heartbeats go through Eth2 on both - there is a crossover with no switch between Eth2 ports (between the servers). For redundancy heartbeats each server checks the other one via Eth2 port, looking for 10.10.10.1 or 10.10.10.2 address accordingly.
The issue: if Server1 goes to reboot then Server2 starts looking for Server1's Eth2 traffic (IP address) through its own Eth1. When Server1 comes back then Server2 never switches back to Eth2, it stays on Eth1. Same thing happens if Server2 goes to reboot.
The system worked fine for a year. Suddenly it stopped going correct way. There were some updates installed for Windows, but I am not too sure which ones because I did not install them.
Does anybody knows what could be the issue why after Server1 reboot the other server still continues to go through Eth1 but not through Eth2?
Thanks in advance.
Update 1: The heartbeat mechanism does not matter.
The system is an Industrial Control System with 2 redundant servers. Eth1 ports on both of the servers are used for data exchange between the control system and the PLCs, while Eth2 ports are used for the monitoring of the system health. Basically, there is a Primary server and a Standby server. The Primary server constantly polls all the data from other devices (PLCs), while the Standby server sits in Standby mode polling only 1 item from each device and ready to become active if needed.
The reason for Eth2 network is to make sure the Backup server is always available and if there is a need (if there are any issues on the Primary server) then the system will automatically switch to the Standby server. I do not know exact heartbeat mechanism on Eth2 network, but Eth2 network exists only for these purposes - to make sure there is a Standby server available if needed.
By design of the system it is required to have 2 networks and they have to be in different networks (even the first octet should be different - 192.x.x.x and 10.x.x.x, for example).
My questions is: why after one of the servers reboots the other server's Eth2 traffic (direct link, no gateway) goes through Eth1 port, not through Eth2?