I am setting up a new network. I have 4 routers in a full mesh. Edge1, Edge2, Client1, Client2.
This are Mikrotik Cloud Core Routers (not that it matters). They are set up with OSPF (distributing connected and static routes) and MPLS, and BGP.
Edge1 has an upstream provider that provides a full BGP feed.
Edge2 has an upstream provider with just a gateway address.
Client1 has a downstream client, will provide full BGP feed to.
Client2 has a downstream client, will provide a partial BGP feed to.
All four routers have a iBGP feed with the other 3 routers. BGP settings for all are the same, redistribute other BGP. Don't redistribute OSPF / Connected / Static. Though I have tried changing those with no success.
That is the basic setup, more specifics below. All four routers can ping all their IP addresses just fine. Any router to any router, perfect (as far as I can tell) network.
Each router can ping it's external Gateway, but other routers in the mesh can not.
For instance.
Edge1's upstream provider
NOC1 - Gateway Address A.B.238.65/29
Edge1's IP addresses A.B.238.66
Edge1 can ping A.B.238.65. Edge2, Client1, Client2 can all ping A.B.238.66 (so they know the route to that lan), but they can not ping A.B.238.65
This goes for every router. Any of the four routers can ping my side of any external lan attached to any of the four routers. Each router that is directly attached to that network can ping the external gateway on that network. But none of the other routers can ping that external gateway.
The second part of this is that I have my IP block I am announcing. It is a /24. We'll call it Z.X.100.0/24. If I announce that block to my BGP upstream guy, I can ping and access all 4 of my routers from the internet. So they can all respond once they have the routes from BGP. BUT they still can't ping the external gateways. The above mentioned problem still exists.
Detailed Configuration (Sorry, they are configured through the UI, so no scripts):
::Shorthand::
R1 -> Edge1
R2 -> Edge2
R3 -> Client1
R4 -> Client2
Edge1:
Eth1ToUpStream - A.B.238.66/29
Eth2ToR2 - 192.168.1.17/30
Eth3ToR3 - 192.168.1.13/30
Eth4ToR4 - 192.168.1.10/30
Loopback - Z.X.100.1/32 <- Z.X.100.0/24 is what I am announcing upstream as well.
OSPF (Just one area):
Redistribute Connected
Redistrubute Static
Connected to all interfaces
Networks:
(Tried both all the specific /30 networks or just the aggregate /24s. Left as /24s)
192.168.1.0/24
Z.X.100.0/24
A.B.238.64/29
MPLS:
Transport Address:
Z.X.100.1 (with LDP to each of the other three in the mesh)
Failure Case:
Try to ping Z.X.100.66 (via Client1/R3) fails
ping to Z.X.100.65 (via Client1/R3) works
Try to ping C.D.176.65 (via Edge2/R2) fails
ping C.D.176.69 (via Edge2/R2) works
Edge2:
Eth1ToUpStream - C.D.176.69/29
Eth2ToR1 - 192.168.1.18/30
Eth3ToR3 - 192.168.1.21/30
Eth4ToR4 - 192.168.1.6/30
Loopback - Z.X.100.2/32 <- Z.X.100.0/24 is what I am announcing upstream as well.
OSPF (Just one area):
Redistribute Connected
Redistrubute Static
Connected to all interfaces
Networks:
(Tried both all the specific /30 networks or just the aggregate /24s. Left as /24s)
192.168.1.0/24
Z.X.100.0/24
C.D.176.64/29
MPLS:
Transport Address:
Z.X.100.2 (with LDP to each of the other three in the mesh)
Failure Case:
Try to ping Z.X.100.66 (via Client1/R3) fails
ping to Z.X.100.65 (via Client1/R3) works
Try to ping A.B.238.66 (via Edge1/R1) fails
ping to A.B.238.65 (via Edge1/R1) works
Client1:
Eth1ToClient - Z.X.100.65/29
Eth2ToR1 - 192.168.1.14/30
Eth3ToR2 - 192.168.1.22/30
Eth4ToR4 - 192.168.1.2/30
Loopback - Z.X.100.3/32 <- Z.X.100.0/24 is what I am announcing upstream as well.
OSPF (Just one area):
Redistribute Connected
Redistrubute Static
Connected to all interfaces
Networks:
(Tried both all the specific /30 networks or just the aggregate /24s. Left as /24s)
192.168.1.0/24
Z.X.100.0/24
MPLS:
Transport Address:
Z.X.100.3 (with LDP to each of the other three in the mesh)
Failure Case:
Try to ping C.D.176.65 (via Edge2/R2) fails
ping C.D.176.69 (via Edge2/R2) works
Try to ping A.B.238.66 (via Edge1/R1) fails
ping to A.B.238.65 (via Edge1/R1) works
Client2:
Eth1ToClient - Z.X.100.69/29
Eth2ToR1 - 192.168.1.9/30
Eth3ToR2 - 192.168.1.5/30
Eth4ToR3 - 192.168.1.1/30
Loopback - Z.X.100.4/32 <- Z.X.100.0/24 is what I am announcing upstream as well.
OSPF (Just one area):
Redistribute Connected
Redistrubute Static
Connected to all interfaces
Networks:
(Tried both all the specific /30 networks or just the aggregate /24s. Left as /24s)
192.168.1.0/24
Z.X.100.0/24
MPLS:
Transport Address:
Z.X.100.4 (with LDP to each of the other three in the mesh)
Failure Case:
Try to ping C.D.176.65 (via Edge2/R2) fails
ping C.D.176.69 (via Edge2/R2) works
Try to ping A.B.238.66 (via Edge1/R1) fails
ping to A.B.238.65 (via Edge1/R1) works
Try to ping Z.X.100.66 (via Client1/R3) fails
ping to Z.X.100.65 (via Client1/R3) works
Additional information. If it helps.
A traceroute from outside the network to Z.X.100.66 fails 192.168.1.14 is the last hop
A traceroute from outside the network to Z.X.100.65 succeeds 192.168.1.14 is not listed in the hops.