Troubleshooting High CPU on 6500 by using Netdr capture and other tools.

To troubleshoot high cpu on Cisco 6500 , follow the below procedure.

When we have issues likes high CPU, we do not have much time to go through all the documents and need only few command to find out the issue and fix it.

1) Check the cpu utilization by using below command and check whether it is due to process or interrupt.

Show process high cpu | e 0.0

PU utilization for five seconds: 99/85%; one minute: Z%; five minutes: W% 
  PID  Runtime(ms)  Invoked  uSecs    5Sec   1Min   5Min TTY Process 

2) If high CPU is  due to interrupt as shown above(85%), then we have following tools to use to troubleshoot.

a) Netdr capture b) inband rp span  c) buffer capture.

Netdr capture is the best way to check what traffic is hitting to  the RP on 6500.

3) Enable the netdr capture by using below command.

debug netdr capture rx   

Note: This command does not have any impact on cpu utilization even when CPU is 99%. We are safe to use this command as I use this command with all my customers without any issues.

4) Use the below command to check the captured packets going to the cpu.

Show netdr captured-packets

Below is the example of collected output by using show netdr captured-packets.

interface Vl80, routine mistral_process_rx_packet_inlin, timestamp 10:13:22.291
dbus info: src_vlan 0x50(80), src_indx 0xB43(2883), len 0x40(64)
  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
  68820400 00500000 0B430100 40080000 00010408 0E000008 00000010 03809CC9
mistral hdr: req_token 0x0(0), src_index 0xB43(2883), rx_offset 0x76(118)
  requeue 0, obl_pkt 0, vlan 0x50(80)
destmac 00.08.E3.FF.FD.90, srcmac 8C.73.6E.C0.7D.80, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 36, identifier 21919
  df 0, mf 0, fo 0, ttl 1, src 134.243.80.203, dst 134.243.80.254
    icmp type 8, code 0

5) As we can see in above output , that gives all the header information as from where the packet is coming and from which IP address and Src address.

6) Use regression expression with show netdr captured-packets to filter the output and find out the percentage of the same kind of packet hitting to the CPU.

Show netdr captured-packets | i interface

Show netdr captured-packets | i srcmac

Show netdr captured-packets | i destmac

Show netdr captured-packets | i src-info 

7)  Once you find out that which interface/src /src vlan you are getting the traffic most, you can either shutdown or use the access-list to block the traffic.

8) If you see that traffic is unicast flow coming to /from particular Ip address, it indicates that traffic is software switching instead hardware switching.

9) You can use the show mls cef command to find out why traffic is not switching in hardware.

  Below are some command which can be use full to get more information on mls cef hardware switching.

  Show ip  interface <gig>| i ip|CEF|UP

  Show fm summary

  show mls cef exception status 

  show mls cef maximum-routes. 

  show ip cef switching statistics 

  show ip cef switching statistics feature 

  show tcam interface <>

I hope above command would help you to give some insight regarding the high CPU troubleshooting.