Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm a novice sysadmin working on a system with a CentOS server masquerading traffic with client machines also running CentOS. Something with the tables got messed up and my clients no longer have internet connectivity.

I'm trying to get ping requests working as a first step and have pinpointed where the traffic flow is failing, but I don't understand the configuration enough to know what to do next. To the more experienced sysadmins, does a quick glance at my tables reveal anything abnormal?


Ping failure.

1. Server.

IP=98.139.183.24 ; _ tcpdump -i any "dst host $IP or src host $IP"

2. Client.

» IP=98.139.183.24; ping $IP
PING 98.139.183.24 (98.139.183.24) 56(84) bytes of data

3. Server.

For normal operation, I should be seeing responses, which makes me suspect something is wrong with the iptable configuration on the master node.

Note: b6 is mapped to 10.0.2.6 in /etc/hosts.

» IP=98.139.183.24 ; _ tcpdump -i any "dst host $IP or src host $IP"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
20:07:38.553547 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 1, length 64
20:07:38.553580 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 1, length 64
20:07:39.552969 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 2, length 64
20:07:39.552983 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 2, length 64
20:07:40.552963 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 3, length 64
20:07:40.552975 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 3, length 64
^C
6 packets captured
6 packets received by filter

Configuration.

Note: I've modified this to the configuration working for me to possibly help anybody with the same issue in the future.

Server.

ip addr

~ » ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 90:e2:ba:21:b8:10 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.0/8 brd 10.255.255.255 scope global p2p1
    inet6 fe80::92e2:baff:fe21:b810/64 scope link 
       valid_lft forever preferred_lft forever
3: p2p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 90:e2:ba:21:b8:11 brd ff:ff:ff:ff:ff:ff
    inet <ext IP>/24 brd <ext ip prefix>.255 scope global p2p2
    inet6 2001:468:c80:2106:92e2:baff:fe21:b811/64 scope global dynamic 
       valid_lft 2591809sec preferred_lft 604609sec
    inet6 fe80::92e2:baff:fe21:b811/64 scope link 
       valid_lft forever preferred_lft forever
4: p2p3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 90:e2:ba:21:b8:14 brd ff:ff:ff:ff:ff:ff
5: p2p4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 90:e2:ba:21:b8:15 brd ff:ff:ff:ff:ff:ff
6: em1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether d4:ae:52:99:8c:29 brd ff:ff:ff:ff:ff:ff
7: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether d4:ae:52:99:8c:2a brd ff:ff:ff:ff:ff:ff

ip route

~ » ip route
<ext ip prefix>.0/24 dev p2p2  proto kernel  scope link  src <ext ip>
<prefix 1>.0.0/16 dev p2p1  scope link  metric 1002 
<prefix 1>.0.0/16 dev p2p2  scope link  metric 1003 
10.0.0.0/8 dev p2p1  proto kernel  scope link  src 10.0.1.0 
default via <ext ip prefix>.1 dev p2p2  proto static

iptables.

~ » _ iptables -t filter -L -v -n
Chain INPUT (policy ACCEPT 3715K packets, 531M bytes)
 pkts bytes target     prot opt in     out     source               destination         
  76M  111G fail2ban-SSH  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 
 151M  183G ACCEPT     all  --  p2p2   *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
  38M 6612M ACCEPT     all  --  p2p1   *       0.0.0.0/0            0.0.0.0/0           
1604K  101M ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
 1923  142K ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 755K   62M ACCEPT     all  --  p2p1   p2p2    0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  p2p2   p2p1    0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 171M packets, 35G bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain fail2ban-SSH (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  76M  111G RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0 

  ~ » _ iptables -t mangle -L -v -n
Chain PREROUTING (policy ACCEPT 1733K packets, 974M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 1722K packets, 973M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 10956 packets, 892K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 1989K packets, 201M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 2001K packets, 202M bytes)
 pkts bytes target     prot opt in     out     source               destination     

~ » _ iptables -t nat -L -v -n
Chain PREROUTING (policy ACCEPT 26992 packets, 6507K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 10234 packets, 954K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 532K   54M  MASQUERADE  all  --  *      p2p2       10.0.0.0/8           0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 3229 packets, 394K bytes)
 pkts bytes target     prot opt in     out     source               destination

ipforward: yes

» cat /proc/sys/net/ipv4/ip_forward 1

» cat /etc/sysctl.conf | grep ip_forward net.ipv4.ip_forward = 1

Client: 10.0.2.6.

ip addr

~ » ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:03:73:0d:89:15 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.6/8 brd 10.0.0.255 scope global em1
    inet6 fe80::1a03:73ff:fe0d:8915/64 scope link 
       valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 18:03:73:0d:89:17 brd ff:ff:ff:ff:ff:ff

ip route.

~ » ip route
<prefix 1>.0.0/16 dev em1  scope link  metric 1002 
10.0.0.0/8 dev em1  proto kernel  scope link  src 10.0.2.6 
default via 10.0.1.0 dev em1

iptables.

» _ iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
~ » _ iptables -t filter -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
~ » _ iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
share|improve this question

3 Answers 3

up vote 1 down vote accepted

As from the output of the tcpdump, the packets are forwarded to the destination but there are no replies. This is no wonder as the rule in your POSTROUTING chain is broken:

Chain POSTROUTING (policy ACCEPT 10234 packets, 954K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 532K   54M            all  --  *      *       10.0.0.0/8           0.0.0.0/0 

The target for the rule is missing. This should probably be "SNAT" or "MASQUERADE". Without this, the source address of the packets will not be rewritten. This will cause the pinged server to reply to 10.0.2.6 (which is not routable) instead of the server IP.

share|improve this answer
    
Okay, thanks! I've set this up as here. After reloading iptables with this new rule, however, I'm still unable to connect with the clients. I've updated the main post to reflect this change. (I've also tried with source = 0.0.0.0 -- no success.) –  highphi Dec 1 '13 at 3:04
    
Ah! Using -o all was my problem. Changing this to -o p2p2 fixed my problem. Thanks so much! –  highphi Dec 1 '13 at 3:11

if i understand correcnly: the problem is twofold

1: the box you want to pat/route thru does not have iptables masquardade configured. on the pat/masq'ing internet facinf router:

iptbales -A PORTROUTING -t NAT -o eth0 -j MASQUARADE
echo 1 > /proc/sys/net/ip4v/ip_forward

2) the clients want all traffic to appear to be sourced from remote masq'ing router, and so all clients must use this internet router as their default gateway. clients can have all traffic appear to source from there, but they need a route telling them to do so,,,, a default route muse be placved on clients telling then to send traffic there\

ip route del default
ip route add default via ip-of-masq'ing-router

share|improve this answer

I can not post comments to request clarification of your question, so I'll put those questions here. In any case, you didn't provide enough information in order to get this thing answered. I suspect you masquerade isn't working, but can not be sure.

First, there is name "b6" in the tcpdump's output, what is that? Since I don't expect that to be name on the Internet it could be your private IP address leaking to the Internet, and if that's true, you MASQUERADE isn't working. In any case, it would be better to use option -n so that IP addresses aren't converted to DNS names which are in this context meaningless.

Next, your iptables listings are incomplete. Use option -v to dump additional information that will help a lot. Because with that option I could be able to confirm or refute the assumption that your private IP addresses are leaking to the Internet. And also, use -n option not to do reverse DNS queries.

Finaly, you removed IP addresses from routing tables and didn't explain you topology. In that way you also removed potential information that can help debug your problem.

And, as a digression, ip command is much better suited for Linux than ifconfig/route combo.

share|improve this answer
    
Thanks for looking at this and for the comments. I've modified my original question to show b6 mapping to the internal IP 10.0.2.6 (yes, this looks wrong) and iptables with -v and -n. I've kept my external ID address removed, but have replaced most of the <removed> portions with <ext ip> and <prefix 1>. Let me know if you need anything else, thanks! –  highphi Nov 30 '13 at 15:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.