iptables drops packets after POSTROUTING

Question

I have a malware analysis environment that will intercept traffic to arbitrary domains and Internet services using InetSim. I have a sandbox that has its DNS server set to the InetSim instance, and InetSim will answer any DNS query with its own IP address.

This setup works well for malware that calls back to a domain name, but if it attempts to connect directly with a hard coded IP address, my malware environment misses it. I'm attempting to use iptables to redirect any outbound traffic back to the InetSim instance on the same subnet, but packets appear to be getting dropped somewhere on the gateway machine.

There are three machines on the network

1. Gateway (Ubuntu 14.04LTS, running VirtualBox with host-only interface vboxnet0 at 192.168.54.1)
2. InetSim (VirtualBox VM, Remnux (Debian) Linux Distro, VBox host-only interface on eth0 at 192.168.54.2)
3. Sandbox (VirtualBox VM, WinXPSP2, VBox host-only interface at 192.168.54.102)

I am generally following the guide outlined in the netfilter NAT documentation, and my iptables rules look like this. Basically the rules are,

1. Outbound traffic (NOT intended for the 192.168.54.0/24 subnet) is sent to the gateway at 192.168.54.1
2. PREROUTING changes the destination address to the InetSim instance at 192.168.54.2
3. POSTROUTING changes the source address to the Gateway at 192.168.54.1

iptable rules

$ sudo iptables -v -t nat -L
Chain PREROUTING (policy ACCEPT 17465 packets, 1818K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   24  1763 LOG        all  --  vboxnet0 any     anywhere            !192.168.54.0/24      LOG level debug prefix "[PREROUTE OUTBOUND]"
   41  2824 DNAT       all  --  vboxnet0 any     anywhere            !192.168.54.0/24      to:192.168.54.2

Chain INPUT (policy ACCEPT 14623 packets, 1341K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 74 packets, 4809 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 73 packets, 4749 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   17   984 LOG        all  --  any    any     192.168.54.0/24      192.168.54.2         LOG level debug prefix "[POSTROUTE Inetsim]"
   41  2513 SNAT       all  --  any    any     192.168.54.0/24      192.168.54.2         to:192.168.54.1

As you can see from the packet counts, the rules are catching traffic. The weird thing is when I run tcpdump on both the gateway machine, and the InetSim machine, I see the rewritten packets from the gateway's capture, but no such packets from the InetSim machine's capture.

Gateway Capture

15:11:28.747298 ARP, Request who-has 192.168.54.1 tell 192.168.54.102, length 46
15:11:28.747305 ARP, Reply 192.168.54.1 is-at 0a:00:27:00:00:00, length 28
15:11:28.747471 IP 192.168.54.102.1041 > 2.3.4.5.80: Flags [S], seq 2061217349, win 65535, options [mss 1460,nop,nop,sackOK], length 0
15:11:28.747513 IP 192.168.54.1.1041 > 192.168.54.2.80: Flags [S], seq 2061217349, win 65535, options [mss 1460,nop,nop,sackOK], length 0

...SYN Repeated 2 more times...

15:11:33.748132 ARP, Request who-has 192.168.54.2 tell 192.168.54.1, length 28
15:11:33.748483 ARP, Reply 192.168.54.2 is-at 08:00:27:c7:4f:7c, length 28

InetSim Capture

10:11:28.649243 ARP, Request who-has 192.168.54.1 tell 192.168.54.102, length 46
10:11:28.649253 ARP, Reply 192.168.54.1 is-at 0a:00:27:00:00:00, length 46
10:11:28.649363 IP 192.168.54.102.1041 > 2.3.4.5.80: Flags [S], seq 2061217349, win 65535, options [mss 1460,nop,nop,sackOK], length 0

...SYN Repeated 2 more times...

10:11:33.650248 ARP, Request who-has 192.168.54.2 tell 192.168.54.1, length 46
10:11:33.650266 ARP, Reply 192.168.54.2 is-at 08:00:27:c7:4f:7c, length 28

I've enabled trace and these are the entries in /var/log/syslog:

kernel: [22504.635493] TRACE: raw:PREROUTING:policy:2 IN=vboxnet0 OUT= MAC=0a:00:27:00:00:00:08:00:27:0b:26:95:08:00 SRC=192.168.54.102 DST=2.3.4.5 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=141 DF PROTO=TCP SPT=1046 DPT=80 SEQ=3347741723 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B401010402) 
kernel: [22504.635504] TRACE: nat:PREROUTING:rule:1 IN=vboxnet0 OUT= MAC=0a:00:27:00:00:00:08:00:27:0b:26:95:08:00 SRC=192.168.54.102 DST=2.3.4.5 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=141 DF PROTO=TCP SPT=1046 DPT=80 SEQ=3347741723 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B401010402) 
kernel: [22504.635508] [PREROUTE OUTBOUND]IN=vboxnet0 OUT= MAC=0a:00:27:00:00:00:08:00:27:0b:26:95:08:00 SRC=192.168.54.102 DST=2.3.4.5 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=141 DF PROTO=TCP SPT=1046 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0 
kernel: [22504.635512] TRACE: nat:PREROUTING:rule:2 IN=vboxnet0 OUT= MAC=0a:00:27:00:00:00:08:00:27:0b:26:95:08:00 SRC=192.168.54.102 DST=2.3.4.5 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=141 DF PROTO=TCP SPT=1046 DPT=80 SEQ=3347741723 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B401010402) 
kernel: [22504.635532] TRACE: filter:FORWARD:policy:1 IN=vboxnet0 OUT=vboxnet0 MAC=0a:00:27:00:00:00:08:00:27:0b:26:95:08:00 SRC=192.168.54.102 DST=192.168.54.2 LEN=48 TOS=0x00 PREC=0x00 TTL=127 ID=141 DF PROTO=TCP SPT=1046 DPT=80 SEQ=3347741723 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B401010402) 
kernel: [22504.635536] TRACE: nat:POSTROUTING:rule:1 IN= OUT=vboxnet0 SRC=192.168.54.102 DST=192.168.54.2 LEN=48 TOS=0x00 PREC=0x00 TTL=127 ID=141 DF PROTO=TCP SPT=1046 DPT=80 SEQ=3347741723 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B401010402) 
kernel: [22504.635538] [POSTROUTE Inetsim]IN= OUT=vboxnet0 SRC=192.168.54.102 DST=192.168.54.2 LEN=48 TOS=0x00 PREC=0x00 TTL=127 ID=141 DF PROTO=TCP SPT=1046 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0 
kernel: [22504.635541] TRACE: nat:POSTROUTING:rule:2 IN= OUT=vboxnet0 SRC=192.168.54.102 DST=192.168.54.2 LEN=48 TOS=0x00 PREC=0x00 TTL=127 ID=141 DF PROTO=TCP SPT=1046 DPT=80 SEQ=3347741723 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B401010402)

All other traffic and connections work as expected, but something is happening in the gateway. Yes, ip_forward is enabled. I know tcpdump sits in the middle of the routing process, and doesn't necessarily capture what's on the wire, so it seems like the packets are getting dropped somewhere between the PREROUTING and POSTROUTING tables. Any help would be greatly appreciated.

Answer 1

The issue here appears to be with how VirtualBox emulates a network interface and/or the network stack. I was able to stand up another VBox Guest, configured as a dedicated gateway, using the same iptables rules, and was able to successfully redirect traffic to an arbitrary IP to my local InetSim instance.

Answer 2

To troubleshoot your setup, strip it down and test each part individually.

I have a firewall server (10.3.1.5), so I added the command to for packets to 1.2.3.4 to an internal box 10.3.1.140 (mil102):

iptables -t nat -A PREROUTING -i eth0 -d 1.2.3.4  -j LOG
iptables -t nat -A PREROUTING -i eth0 -d 1.2.3.4  -j DNAT --to 10.3.1.140

This should be the same as your starting point, and now from an internal machine 10.3.1.129 (hp) I can ping 1.2.3.4. The log on the firewall shows the packets:

Feb  3 15:42:54 firewall kernel: [7052380.506166] IN=eth0 OUT= MAC=00:0c:29:64:b1:4a:00:22:64:f7:b4:b8:08:00 SRC=10.3.1.129 DST=1.2.3.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=27456 DF PROTO=ICMP TYPE=8 CODE=0 ID=922 SEQ=1

And using tcpdump to just display ICMP packets (tcpdump 'icmp[icmptype] = icmp-echo or icmp[icmptype] = icmp-echoreply') I see the packet on mil102 (10.3.1.140):

16:42:55.161125 IP hp > mil102: ICMP echo request, id 922, seq 1, length 64
16:42:55.161185 IP mil102 > hp: ICMP echo reply, id 922, seq 1, length 64

You should be able to get to that point with just the PREROUTING line in the nat table -- verify that it is working before trying the POSTROUTING rule.

I then added the POSTROUTING rules:

iptables -t nat -I POSTROUTING 1 -d 10.3.1.140 -s 10.3.1.0/24 -j LOG
iptables -t nat -I POSTROUTING 2 -d 10.3.1.140 -s 10.3.1.0/24 -j SNAT --to 10.3.1.5

Which changes packets from the local network going to 10.3.1.140 (mil102) to look like they're coming from 10.3.1.5 (firewall).

The log file now shows the ping going through:

Feb  3 15:40:33 firewall kernel: [7052239.848022] IN=eth0 OUT= MAC=00:0c:29:64:b1:4a:00:22:64:f7:b4:b8:08:00 SRC=10.3.1.129 DST=1.2.3.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=21950 DF PROTO=ICMP TYPE=8 CODE=0 ID=32310 SEQ=1
Feb  3 15:40:33 firewall kernel: [7052239.848081] IN= OUT=eth0 SRC=10.3.1.129 DST=10.3.1.140 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=21950 DF PROTO=ICMP TYPE=8 CODE=0 ID=32310 SEQ=1

and my target machine mil102 (10.3.1.140) now shows the source of the pings being the firewall (10.3.1.5):

16:40:35.639037 IP firewall > mil102: ICMP echo request, id 32310, seq 2, length 64
16:40:35.639110 IP mil102 > firewall: ICMP echo reply, id 32310, seq 2, length 64

A few notes about my setup -- eth0 is the internal interface on the firewall, eth1 the external. Both hp and mil102 only have one interface eth0.

My existing nat table has some routing in to -- which is why I used insert commands. This is what my original nat table looked like:

Chain PREROUTING (policy ACCEPT 37 packets, 2362 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1444 73980 DNAT       tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:81 to:10.3.1.129:81

Chain INPUT (policy ACCEPT 18 packets, 1222 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 6 packets, 420 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    5   420 ACCEPT     all  --  *      eth1    0.0.0.0/0            172.20.20.0/24
 116M 7439M MASQUERADE  all  --  *      eth1    0.0.0.0/0            0.0.0.0/0

Please ignore the timestamps on the logs -- I collected data for the example after I had tested everything.

The best advice if this is not working is to simplify the setup until you reach a point where it is working as expected and then add complexity until you either break or reach the goal.