CCIE Troubleshooting: Part 1 (Is it really my fault?)
There are two types of troubleshooting that you'll run into on the CCIE lab:
1. The "Proctor is Evil" Troubleshooting
2. The "Self-Induced" Troubleshooting
The latter type is by far the more time-consuming but also the most important. Basically you messed something up, therefore you have to fix it! (At least if you want the points) The reason it is the most time-consuming is because it could be ANY silly mistake or combination of silly mistakes along the way, and there is no predicting what kinds of things can be done to mess with your own head!
The most important rule with this kind of troubleshooting is time management. Set a time limit of 15 minutes. If you can't figure something out in 15 minutes (no, I don't care how "close" you think you are!) go do something else. Whether this involves a bathroom break, a soda/snack break, standing on your head on the high-quality lab chairs or simply moving on to another "service" or "security" task of your lab makes no difference. The idea is to separate your brain from staring at the same thing over and over.
The longer you stare at something, the more you see what you want and not what's really there. Most self-induced errors are really small, and fairly inane. You know. Those "DUH!" moments once we figure it out. But time management is the consequence we suffer due to silly mistakes. Avoid it!
Anyway... on to the more exciting things. The unpredictable nature of the "Proctor is Evil" Troubleshooting. Having started my training career specializing in the old CIT (Cisco Internetwork Troubleshooting) class, I can greatly appreciate some of the humorous things that MAY get thrown into lab exams. The question becomes, if I don't know what they are and there are many different things that could go wrong... What the heck do I do about it?!?!
Excellent question! Process, my dear. It's all about process.
We've had many different posts and blogs about the things you should be doing in your lab exam. You know... The stuff about reading your exam, re-drawing your diagrams, L2 diagramming, etc. Been there, done that. But then there's troubleshooting. Some things are easy to spot. Others are not! Some things may obvious, others are just plain obnoxious.
The one thing to remember is that during the lab exam, you are there for your proctor's entertainment. As long as this goes through your head, you'll understand why some things are done. It's all in good fun! And once you pass the CCIE Lab exam, it's much easier to laugh about it all!
Start with the basics. IP addressing. You are working on your diagram anyway before your lab. Seriously. Troubleshooting and Diagramming all in one step! This is a good time to quickly verify things; A few different commands to think about, and some differences between them.
1. show ip interface brief | exclude unassigned
2. show interface | include Internet
3. show run | include interface|ip address
Each is slightly different.
Rack1R3(config)#do sh ip int br | exc una
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 145.1.36.3 YES manual up up
FastEthernet0/1 145.1.3.3 YES manual up up
Serial1/2.13 145.1.13.3 YES manual up up
Serial1/3.23 145.1.23.3 YES manual up up
Loopback0 150.1.3.3 YES manual up up
Loopback111 145.1.133.3 YES manual up up
Rack1R3(config)#
This gives me simple IP addresses to check against.
Rack1R3(config)#do sh int | in Internet
Internet address is 145.1.36.3/24
Internet address is 145.1.3.3/24
Internet address is 145.1.13.3/24
Internet address is 145.1.23.3/24
Internet address is 150.1.3.3/24
Internet address is 145.1.133.3/24
Rack1R3(config)#
This gives me IP addresses with VLSM masks that may well be present in my diagram. More information to check. Context can be provided by:
Rack1R3(config)#do sh int | in Internet|is (up|down)
FastEthernet0/0 is up, line protocol is up
Internet address is 145.1.36.3/24
FastEthernet0/1 is up, line protocol is up
Internet address is 145.1.3.3/24
Serial0/0/0 is administratively down, line protocol is down
Serial0/1/0 is administratively down, line protocol is down
Serial0/2/0 is administratively down, line protocol is down
Serial0/2/1 is administratively down, line protocol is down
Serial1/0 is administratively down, line protocol is down
Serial1/1 is administratively down, line protocol is down
Serial1/2 is up, line protocol is up
Serial1/2.13 is up, line protocol is up
Internet address is 145.1.13.3/24
Serial1/3 is up, line protocol is up
Serial1/3.23 is up, line protocol is up
Internet address is 145.1.23.3/24
NVI0 is up, line protocol is up
Loopback0 is up, line protocol is up
Internet address is 150.1.3.3/24
Loopback111 is up, line protocol is up
Internet address is 145.1.133.3/24
Rack1R3(config)#
(I think this is the best option, by the way!)
Rack1R3(config)#do sh run | in interface|ip address
ip telnet source-interface Loopback111
ip ftp source-interface Loopback0
interface Loopback0
ip address 150.1.3.3 255.255.255.0
interface Loopback111
ip address 145.1.133.3 255.255.255.0
interface FastEthernet0/0
ip address 145.1.36.3 255.255.255.0
interface FastEthernet0/1
ip address 145.1.3.3 255.255.255.0
interface Serial0/0/0
no ip address
interface Serial0/1/0
no ip address
interface Serial0/2/0
no ip address
interface Serial0/2/1
no ip address
interface Serial1/0
no ip address
interface Serial1/1
no ip address
interface Serial1/2
no ip address
frame-relay route 112 interface Serial1/3 121
interface Serial1/2.13 point-to-point
ip address 145.1.13.3 255.255.255.0
frame-relay interface-dlci 131
interface Serial1/3
no ip address
frame-relay route 121 interface Serial1/2 112
interface Serial1/3.23 point-to-point
ip address 145.1.23.3 255.255.255.0
frame-relay interface-dlci 132
passive-interface default
no passive-interface FastEthernet0/0
ip nat inside source list 111 interface Loopback0 overload
match ip address 101
match ip address prefix-list NAT-Route
Rack1R3(config)#
This one allows you to see the dotted decimal portion of IPs.
You may do whatever you like the best, but this way you can quickly look at your diagrams and see what's happening. Check to be sure that your routers match your diagram.
Checking IP address only is nice. Checking the IP and Mask is better as you may not "discover" the error until your routing protocol configurations. At that point, you aren't thinking about troubleshooting or IP addresses any longer, so it takes more time to figure things out.
Keep it simple!
The next important stage is the L2 diagramming. Your lab may or may not include any sort of physical connection diagram. While your nice L3 diagram may show a link between SW3 and R2, unless they are physically connected to each other, there may be important steps missing in your functionality! The easiest thing to do is go to your switches and use the "show cdp neighbor" command.
A switch's interfaces are up by default (unlike a routers'). Although, you may have discovered any "down" interfaces by your IP address checks, it's good to see anyway.
Rack1SW1#sh cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - PhoneDevice ID Local Intrfce Holdtme Capability Platform Port ID
Rack1SW4 Fas 0/19 160 R S I WS-C3550- Fas 0/13
Rack1SW3 Fas 0/16 174 S I WS-C3560- Fas 0/13
emanon-3750-Dev Gig 0/2 170 S I WS-C3750- Gig 1/0/2
Rack1R1 Fas 0/1 176 R S I 2811 Fas 0/0
Rack1R3 Fas 0/3 119 R S I 2811 Fas 0/0
Rack1R5 Fas 0/5 129 R S I 2811 Fas 0/0
Rack1SW1#Rack1SW2#sh cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - PhoneDevice ID Local Intrfce Holdtme Capability Platform Port ID
RS.1.1.BB2 Fas 0/24 156 R S I 2811 Fas 0/0
Rack1SW4 Fas 0/19 144 R S I WS-C3550- Fas 0/16
Rack1SW3 Fas 0/16 124 S I WS-C3560- Fas 0/16
Rack1R2 Fas 0/2 157 R S I 2811 Fas 0/0
Rack1R4 Fas 0/4 172 R S I 2811 Fas 0/0
Rack1SW2#Rack1SW3#sh cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - PhoneDevice ID Local Intrfce Holdtme Capability Platform Port ID
RS.1.1.BB3 Fas 0/24 130 R S I 2811 Fas 0/0
Rack1SW4 Fas 0/20 177 R S I WS-C3550- Fas 0/20
Rack1SW4 Fas 0/19 177 R S I WS-C3550- Fas 0/19
Rack1SW2 Fas 0/16 159 R S I WS-C3560- Fas 0/16
Rack1SW1 Fas 0/13 174 R S I WS-C3560- Fas 0/16
Rack1R3 Fas 0/4 163 R S I 2811 Fas 0/1
Rack1R5 Fas 0/6 132 R S I 2811 Fas 0/1
Rack1SW3#Rack1SW4#sh cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - PhoneDevice ID Local Intrfce Holdtme Capability Platform Port ID
Rack1SW2 Fas 0/16 145 R S I WS-C3560- Fas 0/19
Rack1SW3 Fas 0/20 170 S I WS-C3560- Fas 0/20
Rack1SW3 Fas 0/19 170 S I WS-C3560- Fas 0/19
Rack1SW1 Fas 0/13 161 R S I WS-C3560- Fas 0/19
Rack1R4 Fas 0/4 159 R S I 2811 Fas 0/1
Rack1R6 Fas 0/6 165 R S I 3825 Gig 0/1
Rack1SW4#
Draw yourself a quick diagram and center the drawing around your switches. Use colors as needed for trunk vs. access ports. Use colors for ISL vs. 802.1Q trunks. Do whatever makes sense for you. But this too will assist in troubleshooting connections (or lack thereof) later in the lab.
Make sure the interfaces match what you are asked for in your lab. Whether on the diagram for Layer3, or in the lab task list for Layer2 connections. Are ports missing? Some quick "show run interface (intf)" commands may help out there, or at least enough to make a quick note to check later while configuring the trunking section!
Those two discovery items may well handle some of your Troubleshooting tasks, and will certainly save time in later verification or Self-Induced Troubleshooting!
That's enough to whet the appetite a little bit. More on the notorious and/or obnoxious troubleshooting next time. Same Bat time.... Same Bat channel.... ;)
