In Infiniband networks, what is an MGID and how does it relate to multicast group?

Question

I have this error in my Open Subnet manager log.

Sep 22 01:49:53 router3-1 docker[12431]: Sep 22 01:49:53 089580 [68860700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10c37b1f7ebb0001 (MT25408 ConnectX Mellanox Technologies) failed to join non-existing multicast group with MGID ffff:0:ffff:0:180b:1493:fd7f:0, insufficient components specified for implicit create (comp_mask 0x10083)
Sep 22 01:49:57 router3-1 docker[12431]: Sep 22 01:49:57 328091 [6585A700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10bf482fd3ac0001 (MT25408 ConnectX Mellanox Technologies) failed to join non-existing multicast group with MGID ffff:0:ffff:0:1868:c9d6:fd7f:0, insufficient components specified for implicit create (comp_mask 0x10083)
Sep 22 01:50:04 router3-1 docker[12431]: Sep 22 01:50:04 973005 [60850700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10bf482fd3a60001 (MT25408 ConnectX Mellanox Technologies) failed to join non-existing multicast group with MGID ffff:0:ffff:0:9811:7b48:ff7f:0, insufficient components specified for implicit create (comp_mask 0x10083)
Sep 22 01:50:05 router3-1 docker[12431]: Sep 22 01:50:05 291935 [5F04D700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10bf482fd3a40001 (MT25408 ConnectX Mellanox Technologies) failed to join non-existing multicast group with MGID ffff:0:ffff:0:88f3:3bdf:fd7f:0, insufficient components specified for implicit create (comp_mask 0x10083)
Sep 22 01:50:09 router3-1 docker[12431]: Sep 22 01:50:09 699903 [6685C700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10c37b1f7e110001 (MT25408 ConnectX Mellanox Technologies) failed to join non-existing multicast group with MGID ffff:0:ffff:0:787e:40cb:fd7f:0, insufficient components specified for implicit create (comp_mask 0x10083)
Sep 22 01:50:22 router3-1 docker[12431]: Sep 22 01:50:22 389193 [63856700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10bf482fd3f00001 (MT25408 ConnectX Mellanox Technologies) failed to join non-existing multicast group with MGID ffff:0:ffff:0:c867:44ab:fe7f:0, insufficient components specified for implicit create (comp_mask 0x10083)

It occurs quite a bit.

It's a very similar looking problem to the following bug report filed here: https://bugzilla.redhat.com/show_bug.cgi?id=1138851

In that particular report the error is against the NTP mcast group.
i.e. MGID ff12:401b:ffff::101 where somehow 101 is interpreted as 224.0.1.1 and from that they say it's NTP.

How did that 101 become that and what is an MGID?

In my case the MGID shown is: ffff:0:ffff:0:c867:44ab:fe7f:0

Does that translate to 224.0.0.0? it claims to be reserved. What's happening here?

Answer 1

Regarding ff12:401b:ffff::101, we can reference RFC-5156 and see that ff00::/8 are multicast addresses. Looking in RFC-4291, ff12:... is broken down where ff denotes multicast, and 12 is broken into flags 0x1 and scope 0x2. flags 0x1 is "a non-permanently-assigned ("transient" or "dynamically" assigned) multicast address", and scope 0x2 is Link-Local.

The next two groups, 401b:ffff::, define the length of the subnet. Then, according to RFC-3306, the final four bytes are the group ID, in this case 0x101. 0x101 is reserved for NTP, according to RFC-2375. Perhaps not coincidentally, this matches the last two bytes of the IPv4 NTP multicast address, 224.0.1.1(RFC-5771).

An MGID is simply a multicast group ID. You have IP over InfiniBand (IPoIB) configured on your IB interface, and with that, an IP address (Type "ip addr" to list your interfaces and the associated addresses - you're going to get an IPv6 link local address on any interface.) Now, your NTP server is running, multicasting traffic, and opportunisticly trying to use any interface - hence multicasting on your IB network.

Your address is quite a bit different - not a standard IPv6 multicast address, as FFFF:... is reserved. However, usually a quick workaround is to simply restart OpenSM, and the problem goes away.