0

I'm using a Hurricane Electric tunnel on one of my VPSs, and it isn't working completely. I set up the tunnel with a script basically identical to this one: http://www.cybermilitia.net/2013/07/22/ipv6-tunnel-on-openvz/, with only modifications for my particular setup. I can ping the server and fetch a webpage from it, but I get the following output from ping6:

root@unixshell:~# ping6 -c4 2001:470:1f0e:12a7::2
PING 2001:470:1f0e:12a7::2(2001:470:1f0e:12a7::2) 56 data bytes
From 2002:d8da:e02a::1 icmp_seq=1 Destination unreachable: Address unreachable
64 bytes from 2001:470:1f0e:12a7::2: icmp_seq=1 ttl=63 time=96.4 ms
64 bytes from 2001:470:1f0e:12a7::2: icmp_seq=2 ttl=63 time=73.2 ms
From 2002:d8da:e02a::1 icmp_seq=2 Destination unreachable: Address unreachable

--- 2001:470:1f0e:12a7::2 ping statistics ---
2 packets transmitted, 2 received, +2 errors, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 73.256/84.838/96.420/11.582 ms

on the problem server, running tcpdump, I see this simultaneously with the above:

root@tektonic:~# tcpdump -n not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
21:25:44.000024 IP 216.218.224.42 > 207.210.83.205: IP6 2002:cfd2:4a7c::1 > 2001:470:1f0e:12a7::2: ICMP6, echo request, seq 1, length 64
21:25:44.000094 IP 207.210.83.205 > 216.218.224.42: ICMP 207.210.83.205 protocol 41 port 0 unreachable, length 132
21:25:44.000629 IP 207.210.83.205 > 216.218.224.42: IP6 2001:470:1f0e:12a7::2 > 2002:cfd2:4a7c::1: ICMP6, echo reply, seq 1, length 64
21:25:45.020972 IP 216.218.224.42 > 207.210.83.205: IP6 2002:cfd2:4a7c::1 > 2001:470:1f0e:12a7::2: ICMP6, echo request, seq 2, length 64
21:25:45.021059 IP 207.210.83.205 > 216.218.224.42: ICMP 207.210.83.205 protocol 41 port 0 unreachable, length 132
21:25:45.021260 IP 207.210.83.205 > 216.218.224.42: IP6 2001:470:1f0e:12a7::2 > 2002:cfd2:4a7c::1: ICMP6, echo reply, seq 2, length 64
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

And here's the relevant part of my iptables configuration:

root@tektonic:~# iptables --list | egrep '41|ipv6'
ACCEPT     ipv6 --  anywhere             anywhere            
ACCEPT     ipv6 --  anywhere             anywhere

I realize I can just stop sending ICMP unreachable messages using iptables, as mentioned here: Disable ICMP Unreachable replies, however that is a suboptimal solution. Any ideas on how to troubleshoot the actual problem without having to dig through the kernel sources?

The "port 0" part of the error message is a red herring. A 6in4 packet is simply an ipv6 packet with an ipv4 header prepended, and thus has no port number at the ipv4 level. However, the ICMP packets being sent have a type number of 3 and code number 3, meaning "port unreachable", not code 2, "protocol unreachable". Here is one:

12:15:51.011697 IP 207.210.83.205 > 216.218.224.42: ICMP 207.210.83.205 protocol 41 port 0 unreachable, length 132
    0x0000:  45c0 0098 8d6c 0000 4001 0f94 cfd2 53cd  E....l..@.....S.
    0x0010:  d8da e02a 0303 62fb 0000 0000 4500 007c  ...*..b.....E..|
    0x0020:  9157 4000 f829 145c d8da e02a cfd2 53cd  .W@..).\...*..S.
    0x0030:  6000 0000 0040 3a3b 2002 cfd2 4a7c 0000  `....@:;....J|..
    0x0040:  0000 0000 0000 0001 2001 0470 1f0e 12a7  ...........p....
    0x0050:  0000 0000 0000 0002 8000 aa6e 1022 0002  ...........n."..
    0x0060:  ad8a c355 ce94 0a00 0809 0a0b 0c0d 0e0f  ...U............
    0x0070:  1011 1213 1415 1617 1819 1a1b 1c1d 1e1f  ................
    0x0080:  2021 2223 2425 2627 2829 2a2b 2c2d 2e2f  .!"#$%&'()*+,-./
    0x0090:  3031 3233 3435 3637                      01234567

[update 2015-08-06] Upgraded tb_userspace to revision 18, no change.

[update 2015-08-09] tb_userspace.c line 163: sockv6 = socket(AF_INET, SOCK_RAW, IPPROTO_IPV6);, and lsof -c tb_userspace shows the socket is indeed created: tb_usersp 6614 root 4u raw 0t0 1559059549 CD53D2CF:0029->00000000:0000 st=07

[update 2015-08-09 17:18 PDT] confirmed same problem exists on plain kernel without openvz:

jcomeau@unixshell:~$ ping6 2001:470:66:79d::2
PING 2001:470:66:79d::2(2001:470:66:79d::2) 56 data bytes
64 bytes from 2001:470:66:79d::2: icmp_seq=1 ttl=60 time=86.5 ms
From 2001:470:0:206::2 icmp_seq=1 Destination unreachable: Address unreachable
64 bytes from 2001:470:66:79d::2: icmp_seq=2 ttl=60 time=83.4 ms
From 2001:470:0:206::2 icmp_seq=2 Destination unreachable: Address unreachable
64 bytes from 2001:470:66:79d::2: icmp_seq=3 ttl=60 time=86.1 ms
From 2001:470:0:206::2 icmp_seq=3 Destination unreachable: Address unreachable
^C
--- 2001:470:66:79d::2 ping statistics ---
3 packets transmitted, 3 received, +3 errors, 0% packet loss, time 2012ms
rtt min/avg/max/mdev = 83.429/85.376/86.556/1.427 ms

jcomeau@unixshell:~$ logout
Connection to www closed.
jcomeau@aspire:~$ uname -a
Linux aspire 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux

also flushed iptables and ip6tables and removed all netfilter modules. same symptom.

[update 2015-08-11 01:04] from http://linux.die.net/man/7/raw found out that after a raw packet has been dispatched to any raw sockets, the kernel will still pass it to any modules registered for that protocol. the module on my own netbook, which is the "raw kernel without openvz" I was testing on, was tunnel4. once I removed it, the destination unreachable messages stopped. I'm assuming that same module is built into the monolithic kernel on my VPS. /proc/kallsyms does not exist on it, so I will have to contact customer support.

[update 2015-08-11 01:50] http://www.haifux.org/lectures/217/netLec5.pdf is a resource that helped also.

Improve this question
13
  • Where did that 6to4 tunnel come from? That should not be there at all.
    – Michael Hampton
    Aug 6, 2015 at 2:55
  • both servers have a 6to4 tunnel. unfortunately I need them to use ipv6, since my provider doesn't provide it natively.
    – jcomeau_ictx
    Aug 6, 2015 at 3:24
  • 1
    You need to get rid of the 6to4 tunnel if you intend to use Hurricane Electric's tunnel. They will interfere with each other.
    – Michael Hampton
    Aug 6, 2015 at 3:25
  • the only tunnel I have on the problem server, identified as 'tektonic' above, is the Hurricane Electric tunnel.
    – jcomeau_ictx
    Aug 6, 2015 at 3:26
  • 2
    Why use a VPS provider without IPv6 support? There are loads of providers to choose from even if you completely ignore those without IPv6 support.
    – kasperd
    Aug 6, 2015 at 8:55

3 Answers 3

Reset to default
3

as noted in the updates to the question, the problem is that after the kernel passes the packet to whatever raw sockets are listening on that protocol, it then hands it off to any kernel modules registered for that same protocol. since I had been using a sit tunnel on my netbook, the tunnel4 module was still loaded even though I had temporarily set up the tb_userspace tunnel for testing; so since it was registered, but no handlers were configured, it rejected the packets with the ICMP 3:3 message. rmmod sit followed by rmmod tunnel4 solved that problem.

on the original problem server, it wasn't so easy since it's an openvz VPS with a monolithic kernel as seen by the client "boxes". but armed with the information from http://linux.die.net/man/7/raw and http://www.haifux.org/lectures/217/netLec5.pdf I was able to work with the provider to solve the problem. in this case, they re-installed the sit module so I didn't have to use the tb_userspace tunnel software at all. but I suspect the problem was that tunnel4 was installed there as well.

Improve this answer
1

It looks like tektonic does not have the address 2001:470:1f0e:12a7::2 assigned to its venet0 interface. It is receiving the packets and rejecting them even though they are well-formed.

Your next step should be to verify that tektonic can establish TCP connections to IPv6-only hosts such as ipv6.google.com, and that the packets indeed travel to the configured Hurricane Electric relay host via IPv4 encapsulation. If TCP gets through but ICMP does not, then it is definitely an endpoint filtering problem (i.e. firewall rules).

Improve this answer
2
  • note that it is receiving the packets and replying to them despite the ICMP 3:3 message. I added both the tunnel endpoint /64 and the routed /64 to tb0 as the example configuration showed, not to venet0. and I can successfully ping6 ipv6.google.com. the only problem I know of is the ICMP messages being sent.
    – jcomeau_ictx
    Aug 7, 2015 at 2:09
  • Oh, I missed that tektonic is generating IPv6 ECHO_REPLY packets and IPv4 UNREACHABLE packets for the same incoming packets. That's definitely a bug, probably in tb-tun.
    – aecolley
    Aug 7, 2015 at 2:22
1

The ICMP errors are send by the kernel because no socket exists to receive protocol 41 packets with that particular combination of source and destination IP address.

If a process creates a raw socket with protocol 41, then the kernel will stop producing the ICMP errors. Such a socket by default will receive packets from all source IP send to any destination IP assigned to the local machine. Using the bind and/or connect system call, it is possible for the application to restrict which combination of source and destination IP address it will be receiving. Packets which do not match any such socket will still be producing ICMP errors.

It is clear that in your case the packets are simultaneously received by the tunnel and produce ICMP errors. But according to my description above, it is impossible for a packet which is received by a socket to also produce the ICMP error message. But there are other ways the packet could be received, which doesn't stop the kernel from producing ICMP errors.

It is possible for a socket to receive packets at a lower protocol layer where all packets will be visible regardless of IP protocol number. If the tunnel software you are using uses such a low level socket to receive protocol 41 packets, then ICMP errors will be produced like described in your question.

If this is what the tunnel software does, then I would consider it to be a design flaw in the tunnel software. In that case you have three options:

  • Work around the flaw by filtering packets (for instance using iptables).
  • Get the source for the software and fix the design flaw.
  • Switch to different software without this design flaw.
Improve this answer
2
  • the process does create a raw socket for protocol 41. here's the relevant output from lsof: tb_usersp 6614 root 4u raw 0t0 1559059549 CD53D2CF:0029->00000000:0000 st=07
    – jcomeau_ictx
    Aug 9, 2015 at 12:30
  • updated the question to include this.
    – jcomeau_ictx
    Aug 9, 2015 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.