2

I've been using mpong from 29west's mtools package to get some basic idea of multicast latency across various Cisco switches: 1Gb 2960G, 10Gb 4900M and 10Gb Nexus N5548P. The 1Gb is just for comparison.

I have the following results for ~400 runs of mpong on each switch (sending 65536 "ping"-like messages to a receiver which then sends back -- all over multicast). Numbers are latencies measured in microseconds.

Switch           Average      StdDev      Min         Max
2960 (1Gb)       109.68463    0.092816    109.4328    109.9464
4900M (10Gb)     705.52359    1.607976    703.7693    722.1514
NX 5548(10Gb)    58.563774    0.328242    57.77603    59.32207

The result for 4900M is very surprising. I've tried unicast ping and I see the 4900 has ~10us higher latency than the N5548P (average 73us vs 64us). Iperf (with no attempt to tune it) shows both 10Gb switches give me 9.4Gbps line speed.

The two machines are connected to the same switch and we're not doing any multicast routing. OS is RHEL 6. 10Gb NICs are HP 10GbE PCI-E G2 Dual-port NICs (I believe they are rebranded Mellanox cards).

The 4900 switch is used in a project with tight access control so I'm waiting for approval before I can access it and check the config. The other two I have full access to configure.

I've looked at the Cisco document[1] detailing differences between NX-OS and IOS w.r.t multicast so I've got some ideas to try out but this isn't an area where I have much expertise.

Does anyone have any idea what I should be looking at once I get access to the switch?

[1] http://docwiki.cisco.com/wiki/Cisco_NX-OS/IOS_Multicast_Comparison

Edit (12 Jan 0945 GMT):

The 4900M has IGMP snooping enabled. I see no packet loss or errors on the counters on switch or servers.

I've had a look at CPU usage and it seems to sit at 94% when sending the ping messages. 75% is "Cat4k Mgmt LoPri", 6% is "IP Input", 3% is "Cat4k Mgmt HiPri"

Edit2 (12 Jan 1000 GMT):

CPU usage drops to 8% once I stop the messages.

Edit3 (13 Jan 0945 GMT):

Problem is layer 3 related. If I disabling the VLAN interface then latency drops to 72usec.

The config for the vlan is

vlan 110
 name 192.168.110/24-10Ge
end

...snip...

interface Vlan110
 description 10G Test Vlan
 ip address 192.168.110.4 255.255.255.0
 ip pim sparse-mode
end
2
  • perhaps a silly question, but what are the unicast latencies?
    – Joris
    Jan 12, 2011 at 10:11
  • @Joris: unicast latencies are all <1ms Jan 12, 2011 at 10:53

3 Answers 3

1

Dave, your layer 3 prognosis could be right.

try this two documentations: http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

do you have your ip multicast routing enabled? disable this.

enable igmp snooping, enable mrouter - solution 3 (http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008059a9df.shtml#solu1) on switch.

Syson (Toronto)

1
  • I used solution 2 from that secon link you posted and I'm getting the desired results. Many thanks for that! Currently working with the customer to figure if they actually need PIM. Jan 17, 2011 at 11:18
0

Does the 4900 have "ip mcast cef" (or whatever the 'use cef for multicast forwarding' command actually is) enabled? Some platforms will default to using process-switching for multicast forwarding (and some platforms, with some IOS versions will have interesting fault scenarios using CEF for multicast, so....) and that will cause extreme load and latency.

5
  • Not as far as I've been able to find. We've made progress diagnosing it; the problem seems to be a layer3 issue as, if I disable the vlan interface, latency drops down to ~70usec. Jan 13, 2011 at 9:42
  • Might be worth disabling IGMP snooping, I've seen issues due to that on other platforms (though that was packet loss rather than latency).
    – Vatine
    Jan 13, 2011 at 17:18
  • If I disable that then we get the traffic flooded to all switchports. This is undesirable. Jan 14, 2011 at 10:55
  • Might be. Might be worth testing, to see if it is that that's killing your CPU (probably out of hours, though). Can't say I've seen it in the 4900, but I have certainly seen packet-loss due to te presence of IGMP snooping on 4ESW ports.
    – Vatine
    Jan 15, 2011 at 9:45
  • It isn't IGMP snooping which is the problem, it's because of PIM -- but we're not sure why. Disbaling PIM causes traffic to flood to all ports (bad) but latency to drop to a ~60usec (good). Enabling IGMP Snooping Querier keeps latency at 60usec, but it stops traffic flooding to all ports. Jan 17, 2011 at 11:15
0

Solved. The problem was that there wasn't a PIM RP configured.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .