0

Good evening I have a network with few Cisco routers (WS-4900M and 7606(as main core)) on backbone network and different connected to them l3 switches as client access routers, which receives igmp messages for IPTV multicast traffic. It's simple scheme with OSPF+PIM-SM with 7606 as Randezvous Point(with directly connected multicast sources through MSDP+MBGP). [Network scheme][1]

So all clients receive multicast traffic, but on one router(Core2, see scheme) i have a very high CPU usage(80-90% on evenings)(Cat4k Mgmt LoPri), other have less that 10%. When i launched debug platform packet all buffer i received messages like

 Index 2:
54 days 15:56:12:890689 - RxVlan: 1013, RxPort: Te1/8
Priority: High, Tag: No Tag, Event: L3 Forward, Flags: 0x40, Size: 1362
Eth: Src 68:EF:BD:B5:F1:BF Dst 01:00:5E:7F:05:6A Type/Len 0x0800
Ip: ver:IpVersion4 len:20 tos:192 totLen:1344 id:0 fragOffset:0 ttl:19 proto:udp
    src: 172.16.255.2 dst: 239.255.5.106 firstFragment lastFragment

show ip mroute Core2

(*, 239.255.5.106), 01:51:13/00:03:27, RP 172.23.176.252, flags: S
  Incoming interface: TenGigabitEthernet1/8, RPF nbr 172.23.176.69
  Outgoing interface list:
    TenGigabitEthernet1/2, Forward/Sparse, 01:51:13/00:03:27

show ip mroute Core5

(*, 239.255.5.106), 01:51:57/00:03:25, RP 172.23.176.252, flags: S
  Incoming interface: Vlan20, RPF nbr 172.23.180.85
  Outgoing interface list:
    TenGigabitEthernet1/8, Forward/Sparse, 01:51:57/00:02:40, H

show ip mroute Core1

    (*, 239.255.5.106), 06:11:51/00:03:08, RP 172.23.176.252, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Vlan20, Forward/Sparse, 01:52:44/00:03:08

(172.16.255.2, 239.255.5.106), 06:11:51/00:01:36, flags: MT
  Incoming interface: Vlan1532, RPF nbr 10.217.0.130
  Outgoing interface list:
    Vlan20, Forward/Sparse, 01:52:44/00:03:08

So as you see for some reason i have a lot of multicast in buffer and with no (S,G) entry

Then i tried lauch debug ip pim and saw 2 situations With (S,G) entry

   PIM(0): Received v2 Join/Prune on TenGigabitEthernet1/4 from 172.23.177.114, to us
    PIM(0): Join-list: (*, 239.255.2.19), RPT-bit set, WC-bit set, S-bit set
    PIM(0): Update TenGigabitEthernet1/4/172.23.177.114 to (*, 239.255.2.19), Forward state, by PIM *G Join
    PIM(0): Update TenGigabitEthernet1/4/172.23.177.114 to (172.16.2.250, 239.255.2.19), Forward state, by PIM *G Join
    PIM(0): Received v2 Join/Prune on TenGigabitEthernet1/4 from 172.23.177.114, to us
    PIM(0): Join-list: (*, 239.255.2.19), RPT-bit set, WC-bit set, S-bit set
    PIM(0): Update TenGigabitEthernet1/4/172.23.177.114 to (*, 239.255.2.19), Forward state, by PIM *G Join
    PIM(0): Update TenGigabitEthernet1/4/172.23.177.114 to (172.16.2.250, 239.255.2.19), Forward state, by PIM *G Join
    PIM(0): Received v2 Join/Prune on TenGigabitEthernet1/4 from 172.23.177.114, to us
    PIM(0): Join-list: (172.16.2.250/32, 239.255.2.19), S-bit set
    PIM(0): Update TenGigabitEthernet1/4/172.23.177.114 to (172.16.2.250, 239.255.2.19), Forward state, by PIM SG Join
    PIM(0): Received v2 Join/Prune on TenGigabitEthernet1/4 from 172.23.177.114, to us
    PIM(0): Join-list: (172.16.2.250/32, 239.255.2.19), S-bit set
    PIM(0): Update TenGigabitEthernet1/4/172.23.177.114 to (172.16.2.250, 239.255.2.19), Forward state, by PIM SG Join

Without (S,G) entry

PIM(0): Received v2 Join/Prune on TenGigabitEthernet1/2 from 172.23.178.130, to us
PIM(0): Join-list: (*, 239.255.40.232), RPT-bit set, WC-bit set, S-bit set
PIM(0): Check RP 172.23.176.252 into the (*, 239.255.40.232) entry
PIM(0): Building Triggered (*,G) Join / (S,G,RP-bit) Prune message for 239.255.40.232
PIM(0): Add TenGigabitEthernet1/2/172.23.178.130 to (*, 239.255.40.232), Forward state, by PIM *G Join
PIM(0): Building Triggered (*,G) Join / (S,G,RP-bit) Prune message for 239.255.40.232
PIM(0): Insert (*,239.255.40.232) join in nbr 172.23.176.69's queue
PIM(0): Received v2 Join/Prune on TenGigabitEthernet1/2 from 172.23.178.130, to us
PIM(0): Join-list: (*, 239.255.40.232), RPT-bit set, WC-bit set, S-bit set
PIM(0): Update TenGigabitEthernet1/2/172.23.178.130 to (*, 239.255.40.232), Forward state, by PIM *G Join
PIM(0): Building Join/Prune packet for nbr 172.23.176.69
PIM(0):  Adding v2 (172.23.176.252/32, 239.255.40.232), WC-bit, RPT-bit, S-bit Join
PIM(0): Send v2 join/prune to 172.23.176.69 (TenGigabitEthernet1/8)

Have you any ideas for that?

show proc cpu sorted

CPU utilization for five seconds: 44%/4%; one minute: 43%; five minutes: 44%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  85  1833127889  1262153216       1452 32.63% 32.41% 32.89%   0 Cat4k Mgmt LoPri 
 149   237386465  3486312836          0  3.91%  3.91%  3.97%   0 IP Input         
  84    64265071   309949699        207  2.39%  2.30%  2.35%   0 Cat4k Mgmt HiPri 
  94     1932693      236735       8163  0.55%  0.06%  0.00%   0 Per-minute Jobs  
 316      527359     1146186        460  0.15%  0.02%  0.00%   0 SNMP ENGINE      
 327     2415007    33093035         72  0.15%  0.09%  0.07%   0 OSPF-1 Router    
 160          44     4724582          0  0.07%  0.00%  0.00%   0 Socket Timers    
 158         184       45746          4  0.07%  0.00%  0.00%   0 TCP Timer        
 302     1247740   120242829         10  0.07%  0.05%  0.07%   0 PIM Process      
 303        4553    47307586          0  0.07%  0.01%  0.00%   0 Mwheel Process   
 179          48      944992          0  0.07%  0.00%  0.00%   0 Track            
 280        2044       86069         23  0.07%  0.00%  0.00%   0 Syslog           
 107       51538   602283450          0  0.07%  0.08%  0.08%   0 Ethernet Msec Ti 
 320           0         176          0  0.07%  0.00%  0.00%   0 RADIUS           
 282       63499    51998773          1  0.07%  0.01%  0.00%   0 IGMP Input       
 314      158722     3068574         51  0.07%  0.00%  0.00%   0 IP SNMP          
 156           4          47         85  0.07%  0.00%  0.00%   1 Virtual Exec     
  18       21954      139747        157  0.00%  0.00%  0.00%   0 ARP Input        
  17           0           1          0  0.00%  0.00%  0.00%   0 ifIndex Receive  
  20           0           1          0  0.00%  0.00%  0.00%   0 CEF MIB API      
  16           8          12        666  0.00%  0.00%  0.00%   0 RF Slave Main Th 

show platform health

K5CpuMan Review       30.00  58.50     30     14  100  500   69  73   76  60363:49



Packets Dropped In Processing by CPU event

Event             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Sa Miss                              9         0         0         0          0
L2 Router                            5         0         0         0          0
Input ACl Copy                     245         0         0         0          0

Packets Dropped In Processing by Priority

Priority          Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Normal                               5         0         0         0          0
Medium                               9         0         0         0          0
High                               245         0         0         0          0

Packets Dropped In Processing by Reason

Reason             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
STPDrop                               5         0         0         0          0
Tx Mode Drop                        254         0         0         0          0

Total packet queues 64

Packets Received by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Input ACL fwd(snooping)         1388990         0         0         0          0
Host Learning                        9         0         0         0          0
L2 Control                     1117695         0         0         0          0
Input ACL log, unreach         7630372         1         0         1          0
Ttl Expired                    2383808         0         0         0          0
InputIf Fail                      1956         0         0         0          0
Adj SameIf Fail              184833357        47        27        27         23
Bfd                             187937         0         0         0          0
L2 router to CPU, 7           24908694         4         1         2          3
L3 Glean, 7                      31639         0         0         0          0
L3 Fwd, 7                  11851731684      5051      5683      4686       4430
L3 Receive, 7                  2081299         4         0         0          0

Packets Dropped by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Ttl Expired                        118         0         0         0          0
Adj SameIf Fail                  50496         0         0         0          0
L3 Glean, 7                       1365         0         0         0          0
L3 Fwd, 7                      1688513         0         0         0          0

UPD. I just noticed, that Core2 for some reason has active Pim Tunnel0, other routers without it. But i dont know how to disable it. [1]: https://i.stack.imgur.com/sXJlq.png

  • Please post a "show proc cpu". – Overmind Jun 29 at 11:29
  • Updated with it – Maksim Sandakov Jun 29 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.