It is possible, and quite easy to setup. We will use iproute2 and iptables MARK and CONNMARK for this.
The idea is we will mark packets comming in from the second gateway (not the default gateway the server is using), and on reply we will route these packets out the same interface.
Suppose the IP address of the second gateway is 2.2.2.2 and the interface on the server connected to the gateway is eth2.
First let's set up a routing table for the second gateway (we use table 20 for this):
# ip route add default via 2.2.2.2 table 20
And set a rule that says packets marked with 200 will get routed using table 20:
# ip rule add fwmark 200 table 20
You can verify using:
# ip route list table 20
# ip rule list
Now using iptables we mark packets comming in from second gateway (in interface eth2) with mark 200:
1 # iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
2 # iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT
3 # iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 200
4 # iptables -t mangle -A PREROUTING -j CONNMARK --save-mark
For more explanation on how these MARK and CONNMARK work, see here
If 2 of your gateways are on the same network, and your server using only one interface to connect to both of them, then definitely the iptables command number 3 above will not work. You can use another, based on MAC address like this:
# iptables -t mangle -A PREROUTING -m mac --mac-source AA:BB:CC:DD:EE:FF -j MARK --set-mark 200
Of course, AA:BB:CC:DD:EE:FF is the MAC address of the second gateway.
