3

I have multiple Internet routers from the same provider (I'll name “boxes” the Internet routers), all linked to the same hardware network switch forming a physical network (I'll name “NML” that physical network for “No Man's Land”) on which are connected two routers (I'll name them “routers”) doing routing and firewall tasks for a private LAN:

_____________________________________________ WAN (Internet)
    |           |           |           |
    |           |           |           |
 ___|___     ___|___     ___|___     ___|___
[       ]   [       ]   [       ]   [       ]
[ Box 1 ]   [ Box 2 ]   [ Box 3 ]   [ Box 4 ]
[ 0:A:1 ]   [ 0:B:1 ]   [ 0:C:1 ]   [ 0:D:1 ]
[_______]   [_______]   [_______]   [_______]
    |           |           |           |
    |           |           |           |
____|___________|___________|___________|____ NML (A physical network
          |                       |                for boxes and routers)
          |                       |
      ____|____               ____|____ 
     [         ]             [         ]
     [ Router1 ]             [ Router2 ]
     [  0:A:2  ]             [  0:A:3  ]
     [  0:B:2  ]             [  0:B:3  ]
     [  0:C:2  ]             [  0:C:3  ]
     [  0:D:2  ]             [  0:D:3  ]
     [_________]             [_________]
          |                       |
          |                       |
__________|_______________________|__________ LAN (Where people are working)

On the ipv4 side, NML is a local network using an ipv4 private class, it works.

On the ipv6 side, each boxes share the same /64 prefix and each router get an auto-configured ipv6 address from each box (each router get four ipv6, one per box).

To make it simple you can imagine ipv6 addresses with 3 chars:

  • 0:A:1: 0 prefix from provider network prefix, A prefix from first box network prefix, 1 suffix from first box address;
  • 0:A:2: 0 prefix from provider network prefix, A prefix from first box network prefix, 1 suffix from first router getting address from first box;
  • 0:C:1: 0 prefix from provider network prefix, C prefix from third box network prefix, 1 suffix from third box address;
  • 0:C:3: 0 prefix from provider network prefix, C prefix from third box network prefix, 3 suffix from second router getting address from third box.

So each router get four ipv6 addresses, one per box.

  • From router 1 I can ping6 every boxes using their ipv6 addresses (0:A:1, 0:B:1, 0:C:1, 0:D:1);
  • From router 2 I can ping6 every router2 ipv6 addresses (0:A:3, 0:B:3, 0:C:3, 0:D:3), the opposite is true;
  • From both router 1 and router 2 I can't ping6 an ipv6 address, even when I add an explicit route to one box for that ipv6 address (ip -6 route add something via 0:A:1), either using box's ipv6 address from provider prefix either using box's link-local address.

At first only one box had ipv6 activated and at this time I was able to query the Internet using ipv6, but since I activated ipv6 on all the boxes I'm not able to query the Internet using ipv6 anymore. If I do a traceroute6 to an Internet address from a router, it never goes beyond the box.

Note that I don't need to do ipv6 stuff from LAN through the routers at this time, I only need routers to be able to do Internet stuff using ipv6 (mainly to build some VPN over ipv6).

The Internet routers (named “boxes”) are property of ISP and the only option I have is an "enable IPv6" checkbox on the customer page, I don't have any access to router configuration itself and no one other option than enabling or disabling ipv6. The routers between LAN and NML (named “routers”) are standard Debian systems running on some x86-based networking hardware. On a Debian point of view you can imagine it's like a PC : I can do whatever a standard Debian can do.

So, two questions:

  • Is the setup correct and expected to work? If yes, where can I look for to find the problem?
  • Is the setup incorrect and not expected to work at all? If yes, what can I do to fix it?

I removed some "expires" information and stuff like that to remove some verbosity, plus I sometime added some leading 0 to align addresses for easier reading, then replaced some bits to use the ipv6 example class.

# my boxes' ipv6 addresses
box0    2001:db8:ee84:2180::1
box1    2001:db8:ee84:21c0::1
box2    2001:db8:2f13:1ea0::1
box3    2001:db8:399a:08f0::1
box4    2001:db8:399a:39e0::1

# my router1's ipv6 addresses
# ip -6 addr show dev eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2001:db8:ee84:2180:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic
    inet6 2001:db8:ee84:21c0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic
    inet6 2001:db8:2f13:1ea0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:399a:08f0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:399a:39e0:200:24ff:fed1:3d9e/64 scope global mngtmpaddr dynamic 
    inet6 fe80::200:24ff:fed1:3d9e/64 scope link 

# my router2's ipv6 addresses
ip -6 addr show dev eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2001:db8:ee84:2180:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:ee84:21c0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:2f13:1ea0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:399a:08f0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 2001:db8:399a:39e0:200:24ff:fed1:6336/64 scope global mngtmpaddr dynamic 
    inet6 fe80::200:24ff:fed1:6336/64 scope link 

# default ipv6 routes on router1
# ip -6 route
2001:db8:ee84:2180::/64 dev eth2  proto kernel  metric 256
2001:db8:ee84:21c0::/64 dev eth2  proto kernel  metric 256
2001:db8:2f13:1ea0::/64 dev eth2  proto kernel  metric 256
2001:db8:399a:08f0::/64 dev eth2  proto kernel  metric 256
2001:db8:399a:39e0::/64 dev eth2  proto kernel  metric 256
fe80::/64 dev eth2  proto kernel  metric 256 
default via fe80::e69e:12ff:fe04:286f dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe03:8b35 dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe02:10de dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::0224:d4ff:fea7:f258 dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::0224:d4ff:febb:af9e dev eth2  proto ra  metric 1024 hoplimit 64

# default ipv6 routes on router2
# ip -6 route
2001:db8:ee84:2180::/64 dev eth2  proto kernel  metric 256
2001:db8:ee84:21c0::/64 dev eth2  proto kernel  metric 256
2001:db8:2f13:1ea0::/64 dev eth2  proto kernel  metric 256
2001:db8:399a:08f0::/64 dev eth2  proto kernel  metric 256
2001:db8:399a:39e0::/64 dev eth2  proto kernel  metric 256
fe80::/64 dev eth2  proto kernel  metric 256 
default via fe80::e69e:12ff:fe03:8b35 dev eth2  proto ra  metric 1024 mtu 1480 hoplimit 64
default via fe80::0224:d4ff:febb:af9e dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe02:10de dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::e69e:12ff:fe04:286f dev eth2  proto ra  metric 1024 hoplimit 64
default via fe80::0224:d4ff:fea7:f258 dev eth2  proto ra  metric 1024 hoplimit 64

# neigh from router1
# ip -6 neigh | grep eth2
fe80::224:d4ff:febb:af9e dev eth2 lladdr 00:24:d4:bb:af:9e router STALE
fe80::200:24ff:fed1:6336 dev eth2 lladdr 00:00:24:d1:63:36 STALE
fe80::e69e:12ff:fe04:286f dev eth2 lladdr e4:9e:12:04:28:6f router REACHABLE
2001:db8:2f13:1ea0::1 dev eth2 lladdr 00:24:d4:bb:af:9e router STALE
fe80::e69e:12ff:fe02:10de dev eth2 lladdr e4:9e:12:02:10:de router STALE
2001:db8:399a:39e0:200:24ff:fed1:6336 dev eth2 lladdr 00:00:24:d1:63:36 STALE
2001:db8:399a:8f0:200:24ff:fed1:6336 dev eth2 lladdr 00:00:24:d1:63:36 STALE
2001:db8:399a:39e0::1 dev eth2 lladdr 00:24:d4:a7:f2:58 router STALE
2001:db8:399a:8f0::1 dev eth2 lladdr e4:9e:12:02:10:de router STALE
2001:db8:399a:39e0:: dev eth2  FAILED
fe80::213:46ff:fe8f:1e4a dev eth2 lladdr 00:13:46:8f:1e:4a STALE
2001:db8:399a:8f0:: dev eth2  FAILED
fe80::e69e:12ff:fe03:8b35 dev eth2 lladdr e4:9e:12:03:8b:35 router STALE
fe80::224:d4ff:fea7:f258 dev eth2 lladdr 00:24:d4:a7:f2:58 router STALE
fe80::8226:89ff:fe2d:b3d3 dev eth2 lladdr 80:26:89:2d:b3:d3 STALE
fe80::20a:f7ff:fe12:e77 dev eth2 lladdr 00:0a:f7:12:0e:77 STALE
2001:db8:ee84:2180::1 dev eth2 lladdr e4:9e:12:03:8b:35 router STALE
fe80::21d:9ff:fe2c:628d dev eth2 lladdr 00:1d:09:2c:62:8d STALE

# get from router1
# ip -6 route get 2001:4860:4860::8888
2001:4860:4860::8888 from :: via fe80::e69e:12ff:fe04:286f dev eth2  proto ra  src 2001:db8:399a:39e0:200:24ff:fed1:3d9e  metric 1024  hoplimit 64

# neigh from router2
# ip -6 neigh | grep eth2
2001:db8:399a:8f0:200:24ff:fed1:3d9e dev eth2 lladdr 00:00:24:d1:3d:9e STALE
2001:db8:399a:8f0::1 dev eth2 lladdr e4:9e:12:02:10:de router STALE
fe80::e69e:12ff:fe04:286f dev eth2 lladdr e4:9e:12:04:28:6f router STALE
fe80::224:d4ff:fea7:f258 dev eth2 lladdr 00:24:d4:a7:f2:58 router DELAY
fe80::e69e:12ff:fe02:10de dev eth2 lladdr e4:9e:12:02:10:de router REACHABLE
fe80::200:24ff:fed1:3d9e dev eth2 lladdr 00:00:24:d1:3d:9e STALE
fe80::224:d4ff:febb:af9e dev eth2 lladdr 00:24:d4:bb:af:9e router STALE
2001:db8:399a:39e0:200:24ff:fed1:3d9e dev eth2 lladdr 00:00:24:d1:3d:9e STALE
fe80::e69e:12ff:fe03:8b35 dev eth2 lladdr e4:9e:12:03:8b:35 router REACHABLE

# get from router2
ip -6 route get 2001:4860:4860::8888
2001:4860:4860::8888 from :: via 2001:db8:399a:8f0::1 dev eth2  src 2001:db8:399a:39e0:200:24ff:fed1:6336  metric 1024

Some traceroute example:

# traceroute from router1
# traceroute 2001:4860:4860::8888
traceroute to 2001:4860:4860::8888 (2001:4860:4860::8888), 30 hops max, 80 byte packets
 1  2001:db8:ee84:21c0::1 (2001:db8:ee84:21c0::1)  26.146 ms  27.728 ms  28.507 ms
  • So there really are 5 LANs in the same physical LAN. I admit I don't understand why some things aren't symetrical (like why for router2 mtu 1480 on one route and why the route via router's public ip and not link ip), but else, nothing seems missing. I'd tell only the "first" default route should be used, but 2nd router chose the 3rd. No firewall preventing something? And what would give a traceroute -6, or in the other direction a traceroute made from outside ( traceroute6.net ). I hope somebody will now have a clue, because I don't. – A.B Dec 23 '17 at 21:06
  • > So there really are 5 LANs in the same physical LAN. Yes that's what happens there, for the ipv4 stack there is only one lan (common setup), for ip6 there is one lan per box and I have no one option. So if I want to have a dual-stack setup I have to put all the boxes on the same physical network (because of ipv4) which leads to 5 ipv6 lan on the same physical network. I know it's weird but I have no way to control anything on that part, the boxes only have an ipv6 on/off switch. I have to live with it. – Thomas Debesse Dec 24 '17 at 14:56
  • If I remove all the default routes and add only one route (either by public ipv6 box address either link-local box address) it does not work better. If I do a traceroute -6 only the first hop (the default route or the one set for this target) is displayed, and it does not go beyond the router. I have not tried from the outside yet. – Thomas Debesse Dec 24 '17 at 14:59
  • Whatever the problem, you should be using only one box with its ipv6 switch to on until it's working. No need to add complexity, including the description of the issue here. It's no more a "multiple routers" issue if it never worked. – A.B Dec 24 '17 at 15:01
  • As said in the question, it worked when there was only one box with ipv6 activated, it stopped to work once I activated the other boxes. Single box work, multiple boxes do not work. – Thomas Debesse Dec 24 '17 at 15:09
0

I'm transposing this ipv4 example to ipv6.

The trouble is that unless you actively ping each box's ip with the right source, it won't be easy to match (using the mac address) the correct public ip with the correct link local ip. Here's what I could infer from neighbour tables from both routers and from the fact that the link local's end is the mac's end:

box0 2001:db8:ee84:2180::1 <=> fe80::e69e:12ff:fe03:8b35
box2 2001:db8:2f13:1ea0::1 <=> fe80::0224:d4ff:febb:af9e
box3 2001:db8:399a:08f0::1 <=> fe80::e69e:12ff:fe02:10de
box4 2001:db8:399a:39e0::1 <=> fe80::0224:d4ff:fea7:f258

and the remaining one that couldn't be found by the data has to be:

box1 2001:db8:ee84:21c0::1 <=> fe80::e69e:12ff:fe04:286f

With this it becomes clear that both ip -6 route get examples in the question show the wrong source was used for the chosen router. It appears to be a linux' limitation in this case (having multiple LANs overlayed on a single physical LAN).

So here is what has to be done for router1. Also, by trial and error "src" still has to be stated for the default route in the table or an other source might still be chosen. Note that the public IP or the link local IP can (should) be used interchangeably as gateway, so better use the public IP, even if the RA appears to prefer the link ip, it's easier to use the public in a script, and finding the correspondance between the two kinds of IP like above doesn't have to be done anymore.

# ip -6 route add 2001:db8:ee84:2180::/64 dev eth2 src 2001:db8:ee84:2180:200:24ff:fed1:3d9e table 100
# ip -6 route add default via 2001:db8:ee84:2180::1 dev eth2 src 2001:db8:ee84:2180:200:24ff:fed1:3d9e table 100

# ip -6 route add 2001:db8:ee84:21c0::/64 dev eth2 src 2001:db8:ee84:21c0:200:24ff:fed1:3d9e table 101
# ip -6 route add default via 2001:db8:ee84:21c0::1 dev eth2 src 2001:db8:ee84:21c0:200:24ff:fed1:3d9e table 101

# ip -6 route add 2001:db8:2f13:1ea0::/64 dev eth2 src 2001:db8:2f13:1ea0:200:24ff:fed1:3d9e table 102
# ip -6 route add default via 2001:db8:2f13:1ea0::1 dev eth2 src 2001:db8:2f13:1ea0:200:24ff:fed1:3d9e table 102

# ip -6 route add 2001:db8:399a:08f0::/64 dev eth2 src 2001:db8:399a:08f0:200:24ff:fed1:3d9e table 103
# ip -6 route add default via 2001:db8:399a:08f0::1 dev eth2 src 2001:db8:399a:08f0:200:24ff:fed1:3d9e table 103

# ip -6 route add 2001:db8:399a:39e0::/64 dev eth2 src 2001:db8:399a:39e0:200:24ff:fed1:3d9e table 104
# ip -6 route add default via 2001:db8:399a:39e0::1 dev eth2 src 2001:db8:399a:39e0:200:24ff:fed1:3d9e table 104

At least a "default" default route has to be put in the main (the usual) table, else whatever is done later, there will be an error like "no route to host" or "network is unreachable". Since you already got 5 of these from router advertisement, it's not needed.

Now let's add the rules that bind each source IP to its matching table, thus selecting the right source for any of the (box) routers:

# ip -6 rule add from 2001:db8:ee84:2180:200:24ff:fed1:3d9e table 100
# ip -6 rule add from 2001:db8:ee84:21c0:200:24ff:fed1:3d9e table 101
# ip -6 rule add from 2001:db8:2f13:1ea0:200:24ff:fed1:3d9e table 102
# ip -6 rule add from 2001:db8:399a:08f0:200:24ff:fed1:3d9e table 103
# ip -6 rule add from 2001:db8:399a:39e0:200:24ff:fed1:3d9e table 104

It's also possible to force (or even loadbalance with the statistic --mode nth match) a choice using for example ip6tables ... -j MARK ... with ip -6 rule add fwmark ... table 10x

Now it would be better to script this...

There might be an other issue. If a box-router goes offline, the corresponding default route will disappear quickly on the linux system because it's not advertised anymore, but the IP will stay. I don't know then if this IP might still be wrongly chosen to be used on a remaining default route. If this happens, the ip rules added might override the default route to use the now offline box-router, thus defeating the fail-over mechanism of having multiple boxes as routers. I don't have the means to test this.

  • Oh that's interesting, I did that: "ip -6 route add 2001:db8:ee84:2180::/64 dev eth2 src 2001:db8:ee84:2180:200:24ff:fed1:3d9e ; ip -6 route add 2001:4860:4860::8888 via 2001:db8:ee84:2180::1 dev eth2 src 2001:db8:ee84:2180:200:24ff:fed1:3d9e" and I can now ping6 2001:4860:4860::8888 (google's dns) – Thomas Debesse Jan 3 '18 at 12:37
  • ok so there's no need of aditionnal tables or rules and my answer was overkill. now for why it's happening, at least for the local link routes case, I don't see anything in the routes that tell which of the 5 possible local LANs should be chosen over an other so any is possible. I wonder if using the router's public IP (ie: not the link local IP, coming from router advertisement, that would have to be disabled) would be enough to hint about the correct source to use without having to state it – A.B Jan 3 '18 at 13:08
  • for informations these commands: "ip -6 rule add from 2001:db8:399a:39e0:200:24ff:fed1:3d9e table 104" also needs interface "ip -6 rule add from 2001:db8:399a:39e0:200:24ff:fed1:3d9e dev eth2 table 104" btw I'm not yet able to route properly using these tables, but that's progressing. – Thomas Debesse Jan 3 '18 at 16:21
  • so if I delete all the default routes and use one with a source it works, and I can set alternative routes from arbitrary ip6 if I set the correct source – Thomas Debesse Jan 3 '18 at 16:29
  • Now you'll have to test if the fail-over is working: if you set all manual routes but then keep only one box up (or with the ipv6 function enabled) , does the linux system choose the only remaining one or does it fail 80% of the time? and is it actually possible to have many routers for the default route with the same metric when it's added manually (without the "ra" flag) in the same (main) table? – A.B Jan 3 '18 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.