Wednesday, November 4, 2015
Monday, October 19, 2015
Friday, October 16, 2015
Routing Troubleshooting
Problem : Static route is not removing from the routing table even if the connected interface is down.
Cause :
1. Next hop address is reachable through default route.
2. Next hop is reachable through other longest match subnet.
Solution:
a. Use the static route command with connected interface through which next-hop is reachable in combination with next-hop address.
Cause :
1. Next hop address is reachable through default route.
2. Next hop is reachable through other longest match subnet.
Solution:
a. Use the static route command with connected interface through which next-hop is reachable in combination with next-hop address.
Monday, October 12, 2015
Friday, September 4, 2015
Multicast troubleshooting
PIM Multicast
Problem : Multicast source is not getting register with RP.Cause :
Solution:
Problem : RPF failing for the multicast .
Cause :
Solution:
Problem : Multicast is not switching to shortest path tree.
Cause :
Solution:
IGMP Multicast
Problem : Multicast traffic is flooding.
Cause :
1. IGMP snooping disabled
2. IGMP snooping enabled but no mrouter present.
3. IGMP snooping enabled but IGMP snooping table is empty.
Solution:
a) Make sure IGMP snooping is enabled.
b) IGMP querier is configured .
c) Find out if only source is connected in the vlan and no receiver.
d) If any IGMP version mismatch V1 or V2.
Problem : IGMP Multicast is not working where source and receiver are in same vlan.
Cause :
1. No Mrouter configured.
2. No IGMP querier is configured.
3. No PIM sparse-dense mode command creating mrouter port and IGMP querier.
Solution:
a) Enable the igmp querier on layer 2 switch.
b) Configure the "ip pim sparse-dense mode" on SVI.
BGP Troubleshooting/Common Practices
BGP Best practices.
https://nsrc.org/workshops/2013/nsrc-marwan-bgp-ixp/raw-attachment/wiki/Agenda/06_BGP_BCP.pdf
https://www.nanog.org/meetings/nanog43/presentations/Scholl_BCP_Interconnects_N43.pdf
http://www.bdnog.org/v2/bdnog2_paper/prefix-filter-bcp.pdf
http://bgp4all.com/ftp/seminars/SAFNOG1-BGP-BCP.pdf
http://www.infosec.gov.hk/tc_chi/promotion/files/20140801_3.pdf
https://www.nanog.org/meetings/nanog43/presentations/Scholl_BCP_Interconnects_N43.pdf
http://www.bdnog.org/v2/bdnog2_paper/prefix-filter-bcp.pdf
http://bgp4all.com/ftp/seminars/SAFNOG1-BGP-BCP.pdf
http://www.infosec.gov.hk/tc_chi/promotion/files/20140801_3.pdf
BGP Troubleshooting.
Problem: BGP peer is not getting establish between routers.
Cause :
1) Peer ip address is not reachable .
2) port 179 is blocked by firewall or access-list.
3) BGP configuration is not correct like wrong peer address local-as or remote-as, wrong AS number, wrong authentication/MD5 password or wrong update-source loopback.
4) static route is missing for loopback address end to end.
5) TTL=1 for ebgp neigbor. ebgp-multihop is not configured.
6) MD5 authentication has "space" in password.
7)Duplicate router-id between BGP neighbors.
Debugging command :
debug ip bgp
debug ip tcp transaction
debug ip bgp events
Solution:
a) Ensure BGP local and remote AS configuration is correct.
b) MD5 authentication password is correct on both sides without spaces.
c)Verify update-source loopbak and ebgp-multihop.
d) port 179 is allowed in the path boh end via access-list. Ensure firwall is not blocking.
Problem : BGP neighbor ship is flapping and getting reset.
Cause:
1. Keep alive mismatch
2. MTU mismatch.
3. Hellos are stuck in OutQ behind update packets.
4. Remote router rebooting continually (typical with a 3-5 minute BGP peering cycle time)
5. Remote router BGP process unstable, restarting
6.Traffic Shaping & Rate Limiting parameters
7.MTU incorrectly set on links, PMTU discovery disabled on router.
8. Output drops on the interface or congestion on the queue.
9. High CPU on the router or CPU spike on router.
Solution :
a) Make sure you have same keep alive on both routes.
b) Make sure you have same MTU and can ping MTU 1500 with df bit set.
c) If MTU mismatch can not be fixed, can use path-mtu discovery to overcome.
Tip: BGP update packets are packed to the size of the MTU – keepalives and BGP OPEN packets are not packed to the size of the MTU ⇒ Path MTU problem.
Problem : Router is not installing routes in RIB and ignoring.
Cause :
1. Paths that are marked as not synchronized in the show ip bgp longer-prefixes output
2.Paths for which the NEXT_HOP is inaccessible
3.Paths from an external BGP (eBGP) neighbor if the local autonomous system (AS) appears in the AS_PATH
4.If you enabled bgp enforce-first-as and the UPDATE does not contain the AS of the neighbor as the first AS number in the AS_SEQUENCE
5.Paths that are marked as (received-only) in the show ip bgp longer-prefixes output
Solution:
a) Be sure that there is an Interior Gateway Protocol (IGP) route to the NEXT_HOP that is associated with the path.
Problem : Router is not installing routes in BGP Table.
Cause:
1. No router in global routing table for the network command configured in BGP.
2. No matching route in BGP table for the aggregate-command configured in BGP.
3. Same router-ID in routes coming from Router-reflector in Originator-ID attributes.
4. Same router-IS in routes coming from router-reflector from same cluster-ID.
Problem : Router is not advertising route to another BGP peer.
Cause :
1. No matching route in routing table.
2. Filter is discarding route.
Solution :
Problem : Peering changes.
Cause :
Solution:
Problem :Route flapping.
Cause :
Solution:
Problem : Route hijacking.
Cause :
Solution:
Problem : DDoS mitigation
Cause :
Solution:
BGP Reference Materials.
https://tools.ietf.org/html/rfc1998https://transition.fcc.gov/bureaus/pshs/advisory/csric3/CSRIC_III_WG4_Report_March_%202013.pdf
https://www.nanog.org/meetings/nanog51/presentations/Sunday/NANOG51.Talk3.peering-nanog51.pdf
ftp://ftp.registro.br/pub/gter/gter30/TutorialBGP/7%20-%20Transit.pdf
http://ftp.ines.ro/doc/isp-workshops/Classroom%20Modules%203.0/07-bgp-route-filtering.pdf
https://www.nanog.org/meetings/nanog40/presentations/BGPcommunities.pdf
https://www.nanog.org/meetings/nanog41/presentations/BGPMultihoming.pdf
http://conference.apnic.net/__data/assets/pdf_file/0011/58745/apricot2013-bgp-multihoming_1361675367.pdf
Thursday, August 27, 2015
6880 ISSU upgrade / 6800ia ISSU upgrade
IMPORTANT :
===========
Auto image download & image download during "issu runversion fex" will fail,if the "ip tftp source-interface" config is present on the controller.
CSCul10482 TFEX: Image auto download fails due to"ip tftp source-interface" config
RECOVERY STEPS:
================
Remove "ip tftp source-interface" config on IA Parent and add astatic route to copy image through"mgmt" intf.
SUMMARY OF ISSU STEPS :
=======================
Step 1 :
=====
Make sure that the new Cisco IOS image (Cisco IOS software Release 15.1(2)SY1) is present in the bootdisk and the slavebootdisk.
Step 2 :
=====
Use these commands in order to verify that the VSS is ready to run the upgrade procedure:
show issu state detail
show redundancy
show module switch all
6K1#show issu state detail
Step 3 :
=====
Use the issu loadversion command in order to start the upgrade process.
6K1#issu loadversion 1/2 bootdisk:s2t54-adventerprisek9-mz.SPA.151-2.SY1.bin 2/2 slavebootdisk:s2t54-adventerprisek9-mz.SPA.151-2.SY1.bin
In this step, the VSS standby chassis reboots, reloads with the new image, and initializes as the VSS standby chassis in SSO redundancy mode, running the new image. This step is complete when the chassis configuration is
synchronized, as indicated by the Bulk sync succeeded message. It might take several seconds to a few minutes for the new image to load and for the VSS
standby chassis to transition to SSO mode.
Step 4 :
=====
When the VSS standby chassis successfully runs the "new image in the SSO redundancy state and all the LC on the VSS standby chassis are up and online",
enter the issu runversion command in order to force a switchover.
The upgraded VSS standby chassis takes over as the new active chassis, running the new image. The formerly active chassis reloads and initializes as the new VSS
standby chassis in SSO mode, running the old image (in case the software upgrade needs to be aborted and the old image restored). This step is complete
when the chassis configuration is synchronized, as indicated by the Bulk sync succeeded message.
6K1#issu runversion
Step 5 :
=====
Use the issu acceptversion command in order to stop the Rollback Timer. This is necessary because if the timer expires, the upgraded chassis reloads
and reverts to the previous software version.
6K1#issu acceptversion
% Rollback timer stopped. Please issue the 'issu commitversion' command..
Note: Ignore the instruction to run "issu commitversion".
Step 6 :
=====
Use the issu runversion fex all / issue runversion fex <fex id> (for controlled downtime) command in order to start the image download and upgrade
procedure on the FEX (6800IA). The FEX triggers the image download from the new software bundle of the Supervisor2T (here Cisco IOS software Release
15.1(2)SY1). If you use FEX stacks, the master is responsible to extract the image to its members. A TFTP server runs at 192.1.1.1.
6K1#issu runversion fex all
% Successfully initiated 'runversion fex' for Fex IDs: 110.
Use 'show issu state' for more information.
Once all the FEX are upgraded(FEX_UPGRADE_COMPLETE) go to next step.
Step 7 :
=====
In order to continue, enter the issu commitversion command to upgrade the VSS standby chassis and complete the ISSU sequence. The VSS standby chassis reboots, reloads with the new image, and initializes as the VSS standby chassis in the SSO redundancy state, running the new image.
This step is complete when the chassis configuration is synchronized, as indicated by the Bulk sync succeeded message and all the LC on the new VSS-
Standby are up and online.
6K1#issu commitversion
%issu commitversion initiated successfully, upgrade sequence will continue shortly
Step 8 :
=====
In order to verify that the upgrade was successful, use these commands:
show issu state detail
show redundancy
show module switch all
Tuesday, August 25, 2015
Networking Interview Question
OSPF
Q. How many types of area exist in OSPF and types of LSA are allowed ?
Q. Different OSPF packet types ?
Q. If you see how as DRother under show ip ospf neigbor . Whats wrong with the output ?
Q. How OSPF acts as distance vector and link state protocol and when ?. Explain how.
Q. How OSPF avoid loops in the network. Explain how.
Q. Why do we need concept of area and why backbone area is required ?
Q. How OSPF support MPLS TE tunnel ? Explain.
Q. How many routers are recommended in OSPF area and what the limit of routers in OSPF domain?
Q. What is iSPF, BFD, Graceful restarts and BFD in OSPF.
Q. What is the SPF algorithm and how does it work .
Q. What trigger the OSPF SPF in the network and how to avoid and reduce the SPF calculations ?
Q. When OSPF routers use multicast address and unicast address ?
Q. OSPF is layer 2 or layer 3 protocol ?
Q. Best practices to implement OSPF in Data center/Enterprise/WAN ?
Q. What is an OSPF area?
Q. What are the differences between LSA structures of area ?
Q. Troubleshoot the exhange/exstart state with OSPF ?
Q. Can a non-backbone area be connected to another non-backbone area and not to backbone?
Q. We want to redistribute routes into a non-backbone Type-5. What type of area would that be?
Q. What need to be similar in the hello packet in-order to form OSPF adjacency?
Q. In what case OSPF adjacency get stuck in Attempt state??
Q. In what case OSPF adjacency get stuck in Extract state??
Q. Does OSPF form adjacency with the neighbor that are on the different subnet??
Q. OSPF load balancing scenario? There is network 172.16.1.0/24 and there are two path to reach this network. R1 would choose which path??
1. R1--100Mbps---R2----OC3(155Mbps)----R4----192.168.1.0/24
2. R1--100Mbps---R3----100Mbps---------R4----192.1681.0/24
Q. How OSPF Forwarding works and why do we need ?
Q. Options to connect non backbone area to backbone area over another non-backbone area ?
Q. When link-state-request and link-state-update in use on OSPF ?
Q. In scenario of 4 ABR in the network, which ABR will advertise the network ?
Q. With which network type will OSPF establish router adjacencies but not perform the DR/BDR election process?
Q. What is the administrative distance of OSPF?
Q. How OSPF virtual link and Sham link work and where it used in the network ?
Q. What are the different types of router link types ?
Q. What are the different types of bits used in OSPF packet types including V B E ?
EIGRP
Q. EIGRP is distance vector or link state ?Q. How EIGRP avoid loops in the network ?
Q. What is feasible condition in EIGRP ?
Q . What is SIA in EIGRP and how to solve it ?
Q. How reliable mechanism work in EIGRP ?
Q. Where EIGRP use multicast and unicast packet for the neighbor relationship. ?
Q. Best practices to implement in enterprise and Data center.
ISIS
Q. How ISIS scales better then OSPF ?
Q. What is ATT bit and route -leaking and why it is required ?
Q. What is level 1 and level2 in ISIS ?
Q. What is pseudonode/DIS and how it communicates ?
Q. ISIS is distance vector or link state protocol ?
Q. How ISIS prevent loop in the network ?
Q. What is the difference between narrow and wide metrics ?
Q. Describe the ISIS route selection process ?
BGP
Q. What is the difference between EBGP and IBGP neighborship ?
Q. What transport protocol does BGP use?
Q. What is an AS_PATH and what is it used for?
Q. What is a Local Preference?
Q. How can you use BGP for load balancing traffic outbound?
Q. How can you use BGP for load balancing traffic inbound?
Q What is the final tiebreaker in BGP path selection criteria?
Q What are the three well known mandatory attributes? What does a well known mandatory attribute imply?
Q What is BGP conditional-routing ??
Q Explain the BGP route selection process?
Q. How BGP multipath works?
Q. Why IBGP full mesh required ?
Q. BGP is link state or distance vector protocol or path vector protocol ? Explain why ?
Q. When do we use BGP in network ?
Q. When we can use BGP instead of any IGP ? Is it option ?
Q. Types of BGP routing table?
Q. Define various BGP path attributes and different types.
Q. Why weight is different path attributes ?
Q. What is confederation?
Q. What is route reflector and why it is required?
Q. When does BGP use 0.0.0.0 router id?
Q. Does route reflector come in actual path during traffic forwarding?
Q. What is Site of origin SOO?
Q. What is the cost of external and internal BGP routes ?
Q. Can we use local preference outside the autonomous system?
Q. Does it require that BGP router-id should reachable in cloud?
Q. What is recursive lookup in BGP and how it works?
Q. If a static route is advertised in BGP without using update source what will be the next hop address in update?
Q. Define various types of communities and why they are used?
Q. Troubleshoot idle state in BGP ?
Q. How many links can be assigned for load balancing or sharing?
Q. In eBGP I am establishing my neighborship with loopback address but it’s not coming up. Please specify different reasons.
Q. Can we redistribute BGP in IGP? Please explain your answers.
Q. What is cluster id?
Q. Receiving updates from eBGP peer, will the next hop change or not?
Q. Receiving updates from iBGP peer, will the next hop change or not?
Q. How loop prevention mechanism works in BGP.
Q. What will happen if route reflector is not getting proper updates?
Q. What will happen if route reflectors does not synchronize?
Q. What is the advantage of using BGP AS Prepend?
Q. What is BGP PIC?
Q. Use BGP as Link Protection in case of Dual PoP?
Q. How to achieve Inter-AS Communication MP-eBGP?
Q. What can happen if Route Reflector(RR) is not getting proper route updates?
Q. What is route reflector synchronization?
Q. What are the common best practices to connect with ISP through BGP ?
Q. What are the latest trends for the load balancing mechanism with ISP ?
Q What are the best practices to configure BGP ?
Q. How to use BGP as PE-CE backdoor link?
Q. What address can be used for the BGP neiborship ? Interface or Loopback ?
Q. What is remotely triggered black holing and how to triggered ?
Q. What is uRPF ?
Q. What is Hierarchical FIB - BGP-PIC?
Q. BGP Graceful Restart, NSR and NSF ?
Q. BGP Redistribution Vs MPLS, which one you will select?
Q. How to trigger black hole remotely ?
Q. What is the difference/similarities between Route reflector and Route server ?
MPLS
Q. Describe the high level MPLS packet forwarding from end to end ?
Q. Why CEF is required for MPLS and how LFIB is built in MPLS ?
Q. What is RD and RT ?
Q. How MPLS VPN works ?
Q. What is the different between LDP and TDP , RSVP ?
Q. What are the benefits/disadvantages of LDP and RSVP ?
Q. What are the different label distribution protocol ?
Q What are the VPLS / VPWS and EoMPLS, L2TPv3.
Q. What are the routing protocols used between MPLS PE and Customer Edge ?
Q. What is MPLS Fast reroute /Link protection/Node protection ?
Q. What are the benefits of using LDP over RSVP ?
MULTICAST
Q. What are the differences between PIM SM and PIM DM?
Q. What are the different types of Multicast RP ?
Q. How source and shared multicast tree is built between source and Receiver ?
Q. How PIM Bidirectional is different than PIM SM ?
Q Whats are the difference between IGMPv1,v2 and v3 ?
Q. What is IGMP snooping and IGMP ? Are they different or similar ?
Q. What are the different types of flags in PIM ?
Q. What are the different types of PIM packet types used in multicast ?
Q. What are the differences between PIM BSR and Auto RP.
Q Whats are the difference between IGMPv1,v2 and v3 ?
Q. What is IGMP snooping and IGMP ? Are they different or similar ?
Q. What are the different types of flags in PIM ?
Q. What are the different types of PIM packet types used in multicast ?
Q. What are the differences between PIM BSR and Auto RP.
Layer 2 protocol
Q. What are the similarities and differences between STP/RSTP/MSTP ?
Q. How RSTP is converge faster than STP ?
Q. How TCN works in STP and RSTP and MSTP ?
Q. How to troubleshoot the STP loop ?
Q. What are the portfast, uplink fast and backbone fast ?
Q. Best practices to implement STP/RSTP/MSTP ?
Q.What is flexlinks, UDLD, DHCP snooping , ARP inspection.
Q. How to troubleshoot the STP loop ?
Q. What are the portfast, uplink fast and backbone fast ?
Q. Best practices to implement STP/RSTP/MSTP ?
Q.What is flexlinks, UDLD, DHCP snooping , ARP inspection.
TCP/IP
Q. What is TCP Reno, Tahoe and Cubic,BIC,Woodside algorithm and difference among all ?
Q. What is the default TCP congestion algorithm used on Linux and different Windows laptop.
Q. What is Micro burst ?
Q . What is scaling window system and how does it work?
Q. What is Fast retransmit , Fast recovery, Duplicate Acknowledgement , Selective Acknowledgement ?
Q. How TCP handshake and TCP connection termination works ?
Q. Difference between TCP/UDP ?
Q. What is congestion Window, slow start , retransmission, TCP piggybacking , TCP Nagal Algorithm ?
Q. What is VLSM? What routing protocols support VLSM?
Q. Describe the TCP three-way handshake with flags set for each
Q. What is the directed broadcast address of the subnet 192.168.99.20/30?
Q. How many class C networks are included within a network with a subnet mask of
20 bits (/20)?
Q. Describe how ARP resolution works and does it has Layer 3 header ?
Q. Whats are the RARP, Proxy ARP and Gratuitous ARP ?
Q. How does a switch know where to send a packet right after it powers on?
Q. What is TTL and what is it used for?
Q. How many usable host addresses are in a /30? Where would you use this mask length? Can /31 be used?
Q. Which of the following protocols operate on top of TCP ?
Q. What is a difference between Administrative Distance (AD) and a route metric or cost?
Q. Name some networking management/monitoring tools you are aware of.
Q. True or False:SNMP (Simple Network Monitoring Protocol), Syslog and Radius are protocols that can be used for monitoring systems?
Q. How many bits are used to encode TCP or UDP port number?
Q. What is the range of well known ports for TCP and UDP?
Q.What is MSS??
Q.What is default MSS size??
Q.What is TCP Half Close??
Q.What is Delayed Acknowledgement in TCP??
Q.What would be the result when running multiple traceroute from a single linux machine to same destination??
Q.What would be the result when running multiple traceroute from a single windows machine to same destination???
Q. Explain packet flow when you do ping in the following scenario
Host-1----Switch1----Router1----Router2-----Switch2------Host2
Q. How to do the IP super netting for 192.168.0.1.0/24 and 192.168.2.0/24 ?
Q. What will be first packet on the wire when typing www on browse the internet ?
Q. What is the size of the IPv4, Ethernet frame, Ipv6 ?
Q. What is the G/L bit on the ethernet frame and G/I bit ?
Q. How traceroute works on Microsoft and Linux . Please explain the process and each step.
Q. How traceroute works on the links doing load balancing ?
Q. What are the different types of Ipv4 address types which includes TEST-NET in ?
Q. Different types DDOS attacks and mitigation techniques on network ?
Q. What is RACL/Tacacs/Radius ?
IPv6
Q. What are the field in IPv6 header ?
Q. What are the differences between Ipv6/Ipv4.
Q. What are the different types of IPv6 address types.
Q. How address resolution works on Ipv6.
Q. How server boots up with Ipv6 address.?
Q. How are the Ipv6 address are divided ?
Q. How are the Ipv6 address are divided ?
Optical -Layer 1
Q. What are the requirement for the 40gig and 100gig fiber connection ?
Q. How many fiber strand required for 40gig/100gig fiber ?.
Q. Which one is better Multimode or Single mode fiber ? Explain Why?
Q. Explain the core size of the multimode and single mode fiber ?
Fiber are two types : Single Mode and Multimode.
Multi mode Fiber :
OM1 : 62.5 Micron
OM2 : 50 Micron . 3 times more bandwidth than OM1
OM3 : Used for 10 gig, 40 gig , 100 gig.
OM4 : Used for 10 gig, 40 gig , 100 gig.
Note : OM1 and OM2 does not support 40 gig and 100 gig.
For 40 gig : 4 pairs of fiber are required in tx and 4 pairs of fiber for rx.
For 100 gig : 10 pairs of fiber are required in tx and 10 pairs of fiber for rx.
It has 8-5 to 9 Micron core and used for long distance and more than 5 times expensive compare to multi mode fiber.
Hardware
Q. Describe the path of packet flow on 6500 ?
Q. What are the different kind of ASIC used on 6500 Supervisor and Line card ?
Q. What is TCAM and how it works ?
Q. Describe the standard LTL indexes used on 6500 ?
Q .What are the troubleshooting tools available on 6500 ?
Q. How Nexus 7000 is different from 6500 ?
Q. What are the ASIC used on Nexus 3000 ?
Q. What are the ASIC used on Nexus 3000 ?
Troubleshooting
Q. Define the latest troubleshooting done by you or good cases without any help ?
Q. Replace the line card and supervisor on the 6500 during MW without impacting much to all the users ?
Q.
Design
Q. What are the difference between Layer 2 or Layer 3 design ?
Q. Pros and cons of layer 2 Design ?
Q. Pros and cons of layer 3 design ?
Q. Which design is preferred layer 2 or layer 3 ?
Q. What was the last design you did and on which protocol ?
Q. Design network for Data center ?
Q. Different types of topologies used in Data Center , Enterprise and WAN,Campus.
Q. Different types of architecture Model.
Q. Best practice recommendation for data center network.
Q. How to design network with 6000 ports with 10gig capacity ?
Q. What was the last design you did and on which protocol ?
Q. Design network for Data center ?
Q. Different types of topologies used in Data Center , Enterprise and WAN,Campus.
Q. Different types of architecture Model.
Q. Best practice recommendation for data center network.
Q. How to design network with 6000 ports with 10gig capacity ?
SCRIPTING
Q. Difference between For and while loop ?
Q. What is Class and object ?
Q. What object oriented programming ?
Q. Design a code for number vector ?
Q. What is Python and how to use in networking ?
Q. What are the different type of data structures.
Q. What are different types of loops.
Q .What is mutable and immutable ?
Q. What are list, tuple, dictionaries.
Q. How to handle exception and error in python.
Q. What is module /library ,packages.
Q. Design a code for number vector ?
Q. What is Python and how to use in networking ?
Q. What are the different type of data structures.
Q. What are different types of loops.
Q .What is mutable and immutable ?
Q. What are list, tuple, dictionaries.
Q. How to handle exception and error in python.
Q. What is module /library ,packages.
Management Questions
Q. What are your strength and weakness ?
Q. Where do you see yourself in next five years ?
Q. Why do you want to join this role and company ?
Q. Explain your current responsibilities and role ?
Q. How would you handle the angry VP whose network is down ?
Q. How would you handle the conflicts between you and team member ?Keywords: OSPF networking question, OSPF networking, bgp networking interview questions, EIGRP networking question.
Saturday, August 22, 2015
OSPF Troubleshooting/Best practices
OSPF Design/Best practices.
http://ftp.ines.ro/doc/isp-workshops/Routing%20Presentations/3-ospf-for-isps.pdf
https://www.nanog.org/meetings/nanog49/presentations/Sunday/Shamim_Which_Routing_N49.pdf
1) OSPF Troubleshooting flow chart
2) OSPF Networker troubleshooting PPT
Problem :OSPF full neigborship is not coming up between routers
Cause :
1. Hello and Dead timer or Area ID, Authentication password/type/key or Area type are mismatch between router.
2. Trying to build OSPF neigborship on secondary address.
4. OSPF not enabled on correct interface or network command is wrong.
5. Network type is NBMA and no neigbor map configured with broadcast option.
6. High CPU or OSPF packet is dropped by interface due to queuing or high rate or hardware issue from interface to CPU path.
7. Mismatch Subnet mask is configured.
8. "passive interface <> " is configured under "router ospf"for the interface.
9. Mismatch Network type is configured.
10. Router is configured with ip ospf priority 0 on router.
11. Neighborship is getting built over virtual link on sutb area.
OSPF stuck in INIT (one way hello)
Multicast is broken or layer 2 problem.
Access-list is blocking ospf multicast address.
OSPF hello packet getting NAT translated.
Layer 2 is broken.
OSPF stuck in 2-WAY
Normal on ethernet broadcast.
Layer 2 is broken.
All routers are configured with priority 0 so there will not be any election.
OSPF stuck in EXSTART/EXCHANGE
MTU mismatch between neighbor
Duplicate router-ID between routers.
Packet loss can also cause to stuck.
Access-list is blocking unicast communication between router.
OSPF stuck in LOADING
Neigbor is sending bad packet or corrupt packet due to memory.
LS request packet is not accepting by neighbor and ignoring.
Debug command :
debug ip ospf adj
Solution:
a) Make sure hello-dead/ area id, area type,authentication type/password are correct and same.
b) Make sure MTU is same on both router.
c) Make sure neigbor command is configured on remote router with broadcast.
d) Make sure OSPF neigborship build on primary address.
e) Access-list /control plane is not dropping the packet and allowing OSPF multicast and interface ip address communication.
f) Subnet mask should be same on the router.
g) Make sure no corrupted OSPF packet received.
h) Make sure passive interface is not configured under "router ospf".
i) Make sure Virtual -link is not configured over stub area.
Problem : Route is flapping across the network in OSPF.
Cause : Link flapping on the router.
Solution:
a) Use summarization on the router to reduce the impact of flapping route.
b) Use" ip event dampening " on the interface.
Problem : OSPF Route is not learnt in OSPF database.
Cause:
1) Originating router not generating LSA.
2) Routes are redistributed in stub areas.
3) LSA filter-list is configured on ABR.
Solution:
a) Make sure OSPF adjacency is up.
b) Make sure redistribution is used with subnet keyword as well.
e) Do not redistribute external routes in stub area.
f) Make sure LSA are not filtered.
Problem : OSPF Route is not installed in routing table.
Cause :
1) Distribute list is configured.
3)Network type mismatch between router.
4) Adv-router not reachable msg.
5)One side is numbered and the other unnumbered (O, O IA, O E1,O E2).
6)ip addresses are flipped, dual serial (O, O IA, O E1, O E2).
7) Forwarding address is not known or is known via external/static (O E1, O E2) - route s sum and redistribute conn?
8) Backbone area became discontigous (O, OIA, OE1, OE2)
9) OSPF is enabled on secondary but not on primary.
Solution:
a) Make sure distribute-list is configured correctly.
b) Make sure forwarding address is reachable for external route.
c) Make sure forwarding address is not even reachable but also should be learned via inter area and intra area.
c) Make sure network type is correct on both sides.
d) Make sure there is no backbone partition.
Problem : SPF is running constantly on OSPF router and CPU HOG.
Cause:
1) LSA flapping due duplicate router ID/IP address.
2) Constantly Link flapping in an area.
3)All LSA refresh every 30 min (bad!)•4
4)Timers of each LSA get sync (worse!)
5)With group pacing only LSAs that reach max-age get refreshed periodically
Debug:debug ip ospf monitor
show ip ospf stati
show ip ospf database database-sum
Solution:
a) Use sumarization inter area or external routes on the router if too many.
b) Use" ip event dampening " on the interface.
c) Make sure no duplicate router id is on the network.
d) Make sure links are stable on the network. If links can not stay stable, use summarization.
c) Interval is configurable
Problem : OSPF Neighbor is flapping between router.
Cause:
1. Output and input queue drops/error/crc on the interface.
2. Interface is flapping .
3. High CPU on the router.
4. OSPF hello packet is dropping on the platform between interface and CPU.
Solution:
a) Make sure no high CPU.
b) Interface is stable and no packet drops on the interface due to error/crc/input error/input queue/output drops.
c) No packet drops on the line card and platform.
Problem : NSSA ABR not translating Type 7 LSA
Cause:
Solution:
a) hard code the NSSA type 7 translator by using command "area 1 nssa translate type7 always"
Problem: GRE Tunnel OSPF adjacency is flapping
Cause:
1) Tunnel destination are learning over the Tunnel.
Solution :
a) Make sure tunnel destination is not learn over the OSPF adjacency.
Problem : Sequence number mismatch.
Cause :
1. LSA should one of 5 LSAs
2. If LSA is type 5 and the neighbor is associated with a stub area
3. If one of the options change
4. If the state of MS bit is inconsistent with master slave connection
5. If the I-bit is set
6. If the master receives a DBD packet after a dead interval
7. If the requested LSA is not found, then something has gone wrong with the database exchange.
debug ip ospf adjacency
Solution:
Problem: OSPF: Could not allocate router id
Cause:
1)Common new install problem.
2)If no interface up/up with valid ip address.
3)if no ip addresses assigned.
Solution:
1)Configure a loopback with an ip address.
Problem: OSPF unknown routing protocol
Cause:
1)OSPF is not supported on low end platform
2)For 1000 and 1600 routers download plus version
3)800 routes are not supported to run ospf
Problem:OSPF not sending hellos on async interface
Cause:
1)‘async default routing’ is not configured under the interface
Problem: OSPF not redistributing default static route
Cause:
1)Need default-information originate to propagate default
Problem: OSPF-4-ERRRCV msg on the console
Cause:
1)Mismatch area ID, BAD Checksum etc
E bit is 1, Allow externals, option: 0x2(HEX) = 00000010(Bin)
Stub area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1866 opt 0x0 flag 0x3 len 372
E bit is 0, no external allowed, options: 0x0 = 00000000
MC not supported - ospf ignore lsa mospf command
NSSA: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x118 opt 0x8 flag 0x3 len 372
N/P bit is on, options: 0x8 = 00001000
EA not supported yet
Demand circuit : OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1A1E opt 0x20 flag 0x3 len 392
DC bit is negotiated, options: 0x20 = 00100000
OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492
Flag 0x7--> 111 means I(Initial) = 0, M = 1(More), MS = 1(Master)
Flag 0x6 --> 110 not possible
Flag 0x5 --> 101 not possible
Flag 0x4 --> 100 not possible
Flag 0x3 --> 011 means master has more data to send
Flag 0x2 --> 010 means slave has more data to send
Flag 0x1 --> 001 means master has no more data left to send
Flag 0x0 --> 000 means slave has no more data left to send
00 00 I MM
http://www.itcertnotes.com/2011/06/ospf-options-field.html ---All OSPF bits are explained.
http://ftp.ines.ro/doc/isp-workshops/Routing%20Presentations/3-ospf-for-isps.pdf
https://www.nanog.org/meetings/nanog49/presentations/Sunday/Shamim_Which_Routing_N49.pdf
1) OSPF Troubleshooting flow chart
2) OSPF Networker troubleshooting PPT
Problem :OSPF full neigborship is not coming up between routers
Cause :
1. Hello and Dead timer or Area ID, Authentication password/type/key or Area type are mismatch between router.
2. Trying to build OSPF neigborship on secondary address.
4. OSPF not enabled on correct interface or network command is wrong.
5. Network type is NBMA and no neigbor map configured with broadcast option.
6. High CPU or OSPF packet is dropped by interface due to queuing or high rate or hardware issue from interface to CPU path.
7. Mismatch Subnet mask is configured.
8. "passive interface <> " is configured under "router ospf"for the interface.
9. Mismatch Network type is configured.
10. Router is configured with ip ospf priority 0 on router.
11. Neighborship is getting built over virtual link on sutb area.
OSPF stuck in INIT (one way hello)
Multicast is broken or layer 2 problem.
Access-list is blocking ospf multicast address.
OSPF hello packet getting NAT translated.
Layer 2 is broken.
OSPF stuck in 2-WAY
Normal on ethernet broadcast.
Layer 2 is broken.
All routers are configured with priority 0 so there will not be any election.
OSPF stuck in EXSTART/EXCHANGE
MTU mismatch between neighbor
Duplicate router-ID between routers.
Packet loss can also cause to stuck.
Access-list is blocking unicast communication between router.
OSPF stuck in LOADING
Neigbor is sending bad packet or corrupt packet due to memory.
LS request packet is not accepting by neighbor and ignoring.
Debug command :
debug ip ospf adj
Solution:
a) Make sure hello-dead/ area id, area type,authentication type/password are correct and same.
b) Make sure MTU is same on both router.
c) Make sure neigbor command is configured on remote router with broadcast.
d) Make sure OSPF neigborship build on primary address.
e) Access-list /control plane is not dropping the packet and allowing OSPF multicast and interface ip address communication.
f) Subnet mask should be same on the router.
g) Make sure no corrupted OSPF packet received.
h) Make sure passive interface is not configured under "router ospf".
i) Make sure Virtual -link is not configured over stub area.
Problem : Route is flapping across the network in OSPF.
Cause : Link flapping on the router.
Solution:
a) Use summarization on the router to reduce the impact of flapping route.
b) Use" ip event dampening " on the interface.
Problem : OSPF Route is not learnt in OSPF database.
Cause:
1) Originating router not generating LSA.
2) Routes are redistributed in stub areas.
3) LSA filter-list is configured on ABR.
Solution:
a) Make sure OSPF adjacency is up.
b) Make sure redistribution is used with subnet keyword as well.
e) Do not redistribute external routes in stub area.
f) Make sure LSA are not filtered.
Problem : OSPF Route is not installed in routing table.
Cause :
1) Distribute list is configured.
3)Network type mismatch between router.
4) Adv-router not reachable msg.
5)One side is numbered and the other unnumbered (O, O IA, O E1,O E2).
6)ip addresses are flipped, dual serial (O, O IA, O E1, O E2).
7) Forwarding address is not known or is known via external/static (O E1, O E2) - route s sum and redistribute conn?
8) Backbone area became discontigous (O, OIA, OE1, OE2)
9) OSPF is enabled on secondary but not on primary.
Solution:
a) Make sure distribute-list is configured correctly.
b) Make sure forwarding address is reachable for external route.
c) Make sure forwarding address is not even reachable but also should be learned via inter area and intra area.
c) Make sure network type is correct on both sides.
d) Make sure there is no backbone partition.
Problem : SPF is running constantly on OSPF router and CPU HOG.
Cause:
1) LSA flapping due duplicate router ID/IP address.
2) Constantly Link flapping in an area.
3)All LSA refresh every 30 min (bad!)•4
4)Timers of each LSA get sync (worse!)
5)With group pacing only LSAs that reach max-age get refreshed periodically
Debug:debug ip ospf monitor
show ip ospf stati
show ip ospf database database-sum
Solution:
a) Use sumarization inter area or external routes on the router if too many.
b) Use" ip event dampening " on the interface.
c) Make sure no duplicate router id is on the network.
d) Make sure links are stable on the network. If links can not stay stable, use summarization.
c) Interval is configurable
Problem : OSPF Neighbor is flapping between router.
Cause:
1. Output and input queue drops/error/crc on the interface.
2. Interface is flapping .
3. High CPU on the router.
4. OSPF hello packet is dropping on the platform between interface and CPU.
Solution:
a) Make sure no high CPU.
b) Interface is stable and no packet drops on the interface due to error/crc/input error/input queue/output drops.
c) No packet drops on the line card and platform.
Problem : NSSA ABR not translating Type 7 LSA
Cause:
Solution:
a) hard code the NSSA type 7 translator by using command "area 1 nssa translate type7 always"
Problem: GRE Tunnel OSPF adjacency is flapping
Cause:
1) Tunnel destination are learning over the Tunnel.
Solution :
a) Make sure tunnel destination is not learn over the OSPF adjacency.
Problem : Sequence number mismatch.
Cause :
1. LSA should one of 5 LSAs
2. If LSA is type 5 and the neighbor is associated with a stub area
3. If one of the options change
4. If the state of MS bit is inconsistent with master slave connection
5. If the I-bit is set
6. If the master receives a DBD packet after a dead interval
7. If the requested LSA is not found, then something has gone wrong with the database exchange.
debug ip ospf adjacency
Solution:
Problem: OSPF: Could not allocate router id
Cause:
1)Common new install problem.
2)If no interface up/up with valid ip address.
3)if no ip addresses assigned.
Solution:
1)Configure a loopback with an ip address.
Problem: OSPF unknown routing protocol
Cause:
1)OSPF is not supported on low end platform
2)For 1000 and 1600 routers download plus version
3)800 routes are not supported to run ospf
Problem:OSPF not sending hellos on async interface
Cause:
1)‘async default routing’ is not configured under the interface
Problem: OSPF not redistributing default static route
Cause:
1)Need default-information originate to propagate default
Problem: OSPF-4-ERRRCV msg on the console
Cause:
1)Mismatch area ID, BAD Checksum etc
Options
Normal area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492E bit is 1, Allow externals, option: 0x2(HEX) = 00000010(Bin)
Stub area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1866 opt 0x0 flag 0x3 len 372
E bit is 0, no external allowed, options: 0x0 = 00000000
MC not supported - ospf ignore lsa mospf command
NSSA: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x118 opt 0x8 flag 0x3 len 372
N/P bit is on, options: 0x8 = 00001000
EA not supported yet
Demand circuit : OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1A1E opt 0x20 flag 0x3 len 392
DC bit is negotiated, options: 0x20 = 00100000
* * DC EA N/P MC E *
Flags
Useful in debugging, defines I, M and MS bitsOSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492
Flag 0x7--> 111 means I(Initial) = 0, M = 1(More), MS = 1(Master)
Flag 0x6 --> 110 not possible
Flag 0x5 --> 101 not possible
Flag 0x4 --> 100 not possible
Flag 0x3 --> 011 means master has more data to send
Flag 0x2 --> 010 means slave has more data to send
Flag 0x1 --> 001 means master has no more data left to send
Flag 0x0 --> 000 means slave has no more data left to send
00 00 I MM
OSPF Facts
- Highest IP address ABR routes convert the type7 into type 5.
- Default route is not generated by default in area nssa unless "are nssa <> default originate " configured.
- Totally stubby NSSA area generate the default route by default.
- DR/BDR does not support the preempt therefore if DR fails BDR will become DR new BDR will be connected. DR does not become DR even when it is high priority.
- With "ip ospf priority 0" router does not participate in DR/BDR.
- OSPF behaves as distance vector protocol when multiple area in use.
- Highest priority/IP address becomes the DR/BDR.
- OSPF hellos are always send from primary interface.
OSPF Troubleshooting commands
Show ip ospf neighbor
show ip ospf neigbor detail
show ip ospf
show ip ospf database
show ip ospf interface
show ip ospf database database-summary
show ip ospf virtual-link
show ip ospf database self-originate.
show ip ospf database adv-router
show ip ospf statistics
show ip ospf request-list
show ip ospf bad
debug ip ospf adj
debug ip ospf monitor
show ip ospf request-list
show ip ospf bad
debug ip ospf adj
debug ip ospf monitor
Technology Concepts
http://www.iana.org/assignments/ospfv2-parameters/ospfv2-parameters.xhtml -All OSPF Parameter by IANA.http://www.itcertnotes.com/2011/06/ospf-options-field.html ---All OSPF bits are explained.
Tuesday, August 18, 2015
EIGRP Troubleshooting
Common EIGRP troubleshooting scenario causes and solution.
Problem-1#: EIGRP neighbor not coming up.
Cause :
1.AS number mismatch between neighbor.
2.K value mismatch between routers.
3.IP is on different subnets
4.Primary and secondary address on interface mismatch between routers.
5.layer 2 switch/hub connecting different subnets devices in same VLAN. For example VLAN leaking from vlan 1 to vlan2 by using back to back cable.
6. Access-list blocking EIGRP multicast packets on interface.
Commands:
Show ip eigrp neighbors.
Solution:
a) Make sure same AS number is in use.
b) Make sure same K value is in use.
c) Make sure same IP subnets on both side.
d) Make sure Primary to Primary address match on both side and secondary to secondary.
e) Make sure there is no leaking.
Problem-2# EIGRP neighbor is established but not exchanging routes and getting reset.
Cause :
1. Mismatch mask between the neighbor.
2. Hold time getting expired
3. Retry limit exceeded.
4. Vlan leak is happening or cross VLAN domain due to some redundant link or wrong cabling.
Commands:
Show ip eigrp neighbors.
Solution :
a) Make sure there is no mismatch mask between neighbors.
b) Make sure there is no packet loss on ports (End-CPU to End-End CPU)
c) Make sure Unicast rechability is available and can ping end to end.
d) Make sure there is no access-list blocking the communication unicast communication between neighbors.
e) Make sure COPP is not blocking the unicast packet between neighbors.
f) Make sure there is no MTU mismatch between the neighbors.
g) Make sure EIGRP neigobors on the same subnet and no VLAN are leaking to other subnets.
Extra information :
Mismatch mask causes the routing loop in the network causing acknowledge packet not to processed by router.
Problem#-3: Router stuck in active.
Cause :
1.An active route gets stuck for one of the following reasons:
2.Bad or congested links
3.Low router resources, such as low memory or high CPU on the router
4.Long query range
5.Excessive redundancy
6. Access-list /Copp blocking the unicast packet between neighbors.
Commands:
show ip eigrp topology active
Show ip eigrp neigbors
Solution:
a) Use summarization.
b) Use eigrb stub
b) Use Hierarchy design.
Extra information:
The query stops if one of the following occurs:
All queries are answered from all the neighbors.
The end of network is reached.
The lost route is unknown to the neighbors.
Problem#-4 : Routes not advertising to neigbor:
Cause :
1.Access-list with distribute list is misconfigured.
2.Routers has discontigous network.
3.EIGRP split horizon is enabled
Command: Debug ip eigrp
Solution:
a) Make sure access-list allow the networks in distribute list.
b) Make sure "no auto summary" is enabled.
c) Make sure EIGRP split is disabled on Hub router in Hub/spoke topology.
d) OR Use sub-interface with different subnet between Hub/Spoke.
Problem#-5 : Routes are not installing into routing table.
Cause :
1.Auto or manual summarization configured.
2.Higher administrative distance
3.Duplicate router IDs
Solution:
a) More of design issue. Summarization should not happen at two places. Enable the "no auto-summary" or remove the manual summarization.
b) Reduce the administrative distance of eigrp if it is due to higher AD. Routes will shows as inacceible feasible distnace(metric-45300020202)
c)Make sure router-ID are unique for eigrp on each router. If there is duplicate router ID, EIGRP thinks its a loop and does not install the route in the table.
Extra information:
[Every time autosummarization or manual summarization takes place, EIGRP installs the summary route with the next hop to Null 0. This is a loop-prevention mechanism for EIGRP's summary routes. In this case study, this is exactly what happens—EIGRP does not install a route from its neighbor that falls within its summary range. ]
Saturday, April 4, 2015
Password recovery for Supervisor 2T/6807 chassis.
Step-by-Step Procedure
1. Attach a terminal or PC with terminal emulation to the console port of the router. Use these terminal settings:
9600 baud rate
No parity
8 data bits
1 stop bit
No flow control
2. Powered off and power on switch. Press Break on the terminal keyboard right after the SUP2T/6807-XL switch boots up. If you don't have the break key on the laptop use "break" sequence on PUTTY. From this point on, the password recovery procedure is the same as for any other router.
Recommendation. Always use "Windows laptop" with Putty. MACBOOK does not work for password recovery as it does not send the break sequence and waste lots of time.
Note: If you have dual supervisor , remove one supervisor before power off/on switch otherwise when you will change the config-register value and reset, switch will boot up with other superviosr where you did not change config-register vlaue.
3.Type confreg 0x2142 at the rommon 1> prompt to boot from Flash without loading the configuration.
4. Type reset at the rommon 2> prompt.
5. The switch reboots. However, it ignores the saved configuration.
6. Type no after each setup question or press Ctrl-C to skip the initial setup procedure.
7. Type enable at the Switch> prompt.
8. Issue the configure memory or copy start running commands to copy the Nonvolatile RAM (NVRAM) into memory. Do not issue the configure terminal command.
9. The prompt is now switch(config)#. Issue the enable secret < password > command in global configuration mode to change the enable password.
10. Issue the config-register 0x2102 command, or the value you recorded in Step 2 in global configuration mode (Switch(config)#) to set the configuration value back to its original value.
1. Attach a terminal or PC with terminal emulation to the console port of the router. Use these terminal settings:
9600 baud rate
No parity
8 data bits
1 stop bit
No flow control
2. Powered off and power on switch. Press Break on the terminal keyboard right after the SUP2T/6807-XL switch boots up. If you don't have the break key on the laptop use "break" sequence on PUTTY. From this point on, the password recovery procedure is the same as for any other router.
Recommendation. Always use "Windows laptop" with Putty. MACBOOK does not work for password recovery as it does not send the break sequence and waste lots of time.
Note: If you have dual supervisor , remove one supervisor before power off/on switch otherwise when you will change the config-register value and reset, switch will boot up with other superviosr where you did not change config-register vlaue.
3.Type confreg 0x2142 at the rommon 1> prompt to boot from Flash without loading the configuration.
4. Type reset at the rommon 2> prompt.
5. The switch reboots. However, it ignores the saved configuration.
6. Type no after each setup question or press Ctrl-C to skip the initial setup procedure.
7. Type enable at the Switch> prompt.
8. Issue the configure memory or copy start running commands to copy the Nonvolatile RAM (NVRAM) into memory. Do not issue the configure terminal command.
9. The prompt is now switch(config)#. Issue the enable secret < password > command in global configuration mode to change the enable password.
10. Issue the config-register 0x2102 command, or the value you recorded in Step 2 in global configuration mode (Switch(config)#) to set the configuration value back to its original value.
Subscribe to:
Posts (Atom)