I've been using mpong from 29west's mtools package to get some basic idea of multicast latency across various Cisco switches: 1Gb 2960G, 10Gb 4900M and 10Gb Nexus N5548P. The 1Gb is just for comparison.

I have the following results for ~400 runs of mpong on each switch (sending 65536 "ping"-like messages to a receiver which then sends back -- all over multicast). Numbers are latencies measured in microseconds.

Switch           Average      StdDev      Min         Max
2960 (1Gb)       109.68463    0.092816    109.4328    109.9464
4900M (10Gb)     705.52359    1.607976    703.7693    722.1514
NX 5548(10Gb)    58.563774    0.328242    57.77603    59.32207

The result for 4900M is very surprising. I've tried unicast ping and I see the 4900 has ~10us higher latency than the N5548P (average 73us vs 64us). Iperf (with no attempt to tune it) shows both 10Gb switches give me 9.4Gbps line speed.

The two machines are connected to the same switch and we're not doing any multicast routing. OS is RHEL 6. 10Gb NICs are HP 10GbE PCI-E G2 Dual-port NICs (I believe they are rebranded Mellanox cards).

The 4900 switch is used in a project with tight access control so I'm waiting for approval before I can access it and check the config. The other two I have full access to configure.

I've looked at the Cisco document[1] detailing differences between NX-OS and IOS w.r.t multicast so I've got some ideas to try out but this isn't an area where I have much expertise.

Does anyone have any idea what I should be looking at once I get access to the switch?

[1] http://docwiki.cisco.com/wiki/Cisco_NX-OS/IOS_Multicast_Comparison

Edit (12 Jan 0945 GMT):

The 4900M has IGMP snooping enabled. I see no packet loss or errors on the counters on switch or servers.

I've had a look at CPU usage and it seems to sit at 94% when sending the ping messages. 75% is "Cat4k Mgmt LoPri", 6% is "IP Input", 3% is "Cat4k Mgmt HiPri"

Edit2 (12 Jan 1000 GMT):

CPU usage drops to 8% once I stop the messages.

Edit3 (13 Jan 0945 GMT):

Problem is layer 3 related. If I disabling the VLAN interface then latency drops to 72usec.

The config for the vlan is

vlan 110
 name 192.168.110/24-10Ge


interface Vlan110
 description 10G Test Vlan
 ip address
 ip pim sparse-mode
  • perhaps a silly question, but what are the unicast latencies? – Joris Jan 12 '11 at 10:11
  • @Joris: unicast latencies are all <1ms – davedavedave Jan 12 '11 at 10:53

Dave, your layer 3 prognosis could be right.

try this two documentations: http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

do you have your ip multicast routing enabled? disable this.

enable igmp snooping, enable mrouter - solution 3 (http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008059a9df.shtml#solu1) on switch.

Syson (Toronto)

  • I used solution 2 from that secon link you posted and I'm getting the desired results. Many thanks for that! Currently working with the customer to figure if they actually need PIM. – davedavedave Jan 17 '11 at 11:18

Does the 4900 have "ip mcast cef" (or whatever the 'use cef for multicast forwarding' command actually is) enabled? Some platforms will default to using process-switching for multicast forwarding (and some platforms, with some IOS versions will have interesting fault scenarios using CEF for multicast, so....) and that will cause extreme load and latency.

  • Not as far as I've been able to find. We've made progress diagnosing it; the problem seems to be a layer3 issue as, if I disable the vlan interface, latency drops down to ~70usec. – davedavedave Jan 13 '11 at 9:42
  • Might be worth disabling IGMP snooping, I've seen issues due to that on other platforms (though that was packet loss rather than latency). – Vatine Jan 13 '11 at 17:18
  • If I disable that then we get the traffic flooded to all switchports. This is undesirable. – davedavedave Jan 14 '11 at 10:55
  • Might be. Might be worth testing, to see if it is that that's killing your CPU (probably out of hours, though). Can't say I've seen it in the 4900, but I have certainly seen packet-loss due to te presence of IGMP snooping on 4ESW ports. – Vatine Jan 15 '11 at 9:45
  • It isn't IGMP snooping which is the problem, it's because of PIM -- but we're not sure why. Disbaling PIM causes traffic to flood to all ports (bad) but latency to drop to a ~60usec (good). Enabling IGMP Snooping Querier keeps latency at 60usec, but it stops traffic flooding to all ports. – davedavedave Jan 17 '11 at 11:15

Solved. The problem was that there wasn't a PIM RP configured.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.