I am having a weird problem which I cant figure out - so I was hoping someone here could give me a hand.

First off the end goal is that a specific server in my network runs an IPSEC connection to another company and I want all other servers to route traffic for the IP on that network through this single server.

Server 1 in this example is the server that runs the IPSEC connection. (CentOS 6.6)

Server 2 in this example is an app server that would route traffic for only that specific IP through server 1. (CentOS 6.5)

Some IP's that will be used below:

Server 1

Server 1 Public IP: x.x.x.x
Server 1 Public Broadcast: x.x.x.y
Server 1 Public Gateway: x.x.x.z
Server 1 Internal IP: 10.0.64.10/24

Server 2

Server 2 Public IP: y.y.y.y
Server 2 Public Broadcast: y.y.y.z
Server 2 Public Gateway: y.y.y.a
Server 2 Internal IP: 10.0.64.150/24

Those servers have full connectivity between them internally (i.e. I can ping, ssh etc from one to the other without problem). They also both have full acceess to the internet and can be reached that way


Server 1

Here is an ip a for that

# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:99:12:85 brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/28 brd x.x.x.y scope global eth0
    inet6 xxxx:xxxx:xxxx:xxxx/64 scope link
       valid_lft forever preferred_lft forever
3: eth1:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:99:12:8f brd ff:ff:ff:ff:ff:ff
    inet 10.0.64.10/24 brd 10.0.64.255 scope global eth1
    inet6 fe80::20c:29ff:fe99:128f/64 scope link
       valid_lft forever preferred_lft forever

Here is an ip route

# ip route
x.x.x.y/28 dev eth0  proto kernel  scope link  src x.x.x.x
10.0.64.0/24 dev eth1  proto kernel  scope link  src 10.0.64.10
169.254.0.0/16 dev eth0  scope link  metric 1002
169.254.0.0/16 dev eth1  scope link  metric 1003
default via x.x.x.z dev eth0

Here is a sysctl -p

# sysctl -p
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.default.proxy_arp = 1
net.ipv4.conf.all.rp_filter = 1
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 1

Server 2

I've added a single test ip (8.8.8.8) to server two to test if it works before bringing IPSEC into the equation

Here is an ip a

# ip a
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:0c:29:15:8b:01 brd ff:ff:ff:ff:ff:ff
    inet y.y.y.y/29 brd y.y.y.z scope global eth0
    inet6 fe80::20c:29ff:fe15:8b01/64 scope link
       valid_lft forever preferred_lft forever
3: eth1:  mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:0c:29:15:8b:0b brd ff:ff:ff:ff:ff:ff
    inet 10.0.64.150/24 brd 10.0.64.255 scope global eth1
    inet6 fe80::20c:29ff:fe15:8b0b/64 scope link
       valid_lft forever preferred_lft forever

Here is an ip route

# ip route
8.8.8.8 via 10.0.64.10 dev eth1
y.y.y.z/29 dev eth0  proto kernel  scope link  src y.y.y.y
10.0.64.0/24 dev eth1  proto kernel  scope link  src 10.0.64.150
default via y.y.y.a dev eth0

Now when I try do a ping from Server 2 -> 8.8.8.8 here are the tcpdumps from each server:

Server 2

If I tcpdump on eth0 i get no matches (so the route appears right!). eth1 gets matches:

# tcpdump -vvv -i eth1 -n host 8.8.8.8
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
11:25:55.609902 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 1, length 64
11:25:56.609262 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 2, length 64

Server 1 (The hopeful gateway for 8.8.8.8)

On eth1 (Private)

# tcpdump -vv -i eth1 -n host 8.8.8.8
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes

11:27:20.608766 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 86, length 64
11:27:21.608738 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 87, length 64

On eth0 (public)

# tcpdump -vv -i eth0 -n host 8.8.8.8
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:29:04.608773 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 190, length 64
11:29:05.608800 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 191, length 64

I've disabled the FW on both (as a test), made sure to not have any blocking rules on FORWARD traffic (as a separate test) and I just never get my traffic through from Server 2 to 8.8.8.8. I've also tried substituting 8.8.8.8 for another server that is reachable from both servers and the same thing happens.

I'm open to any suggestions - i'm super confused :)

Thanks in advance, Ian

This is resolved!

I was missing the following rule in my IPTables ruleset:

iptables -t nat -I POSTROUTING 1 -o eth0 --jump MASQUERADE

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.