"Broken MPLS? Troubleshoot it we should, mmmmm." ... yoda

Cheers from London! I learned this week that a "Christmas Cracker" is not a food item. There is so much to know. I am grateful for students willing to show me the ropes here in the UK. Thank you all. Now on to the topic at hand.

MPLS is an important part of the RS Bootcamp, including troubleshooting MPLS.

Here is an MPLS troubleshooting scenario, that has 1 (one,одну,un,uno) configuration issue. Can you spot it? Lets get to it! Here is the diagram.

mpls-ine-blog

Problem: Clients on the 5.5.5.0 network are not able to ping the server, or any other devices, on the 1.1.1.0 network. Your challenge, based on the provided IOS show commands only, is to identify the 1 configuration problem that is causing the network failure.

For this scenario, the PC and Server configurations are correct, as well as all the layer 2 switching infrastructure on the Catalyst switches. R1 and R5 are CE routers, R2 and R4 are PE routers, and R3 is a P router.

We will work from the diagram, using the show commands from right to left, starting with R5. (I have grouped the show commands together, then the individual commands and their output. This way, the post may be used as a study/reference tool.)

R5
show ip int brief | ex una
show ip route
show ip protocols
ping 136.1.45.255

R5#show ip int brief | ex una

Interface                  IP-Address      OK? Method Status                Protocol

FastEthernet0/0            5.5.5.5         YES manual up                    up

FastEthernet0/1            136.1.45.5      YES NVRAM  up                    up

Loopback0                  150.1.5.5       YES NVRAM  up                    up

R5#show ip route

Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

136.1.0.0/24 is subnetted, 2 subnets
R 136.1.12.0 [120/1] via 136.1.45.4, 00:00:08, FastEthernet0/1
C 136.1.45.0 is directly connected, FastEthernet0/1
1.0.0.0/24 is subnetted, 1 subnets
R 1.1.1.0 [120/1] via 136.1.45.4, 00:00:08, FastEthernet0/1
5.0.0.0/24 is subnetted, 1 subnets
C 5.5.5.0 is directly connected, FastEthernet0/0
150.1.0.0/24 is subnetted, 1 subnets
C 150.1.5.0 is directly connected, Loopback0
R5#show ip protocols
Routing Protocol is "rip"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Sending updates every 30 seconds, next due in 11 seconds
Invalid after 180 seconds, hold down 180, flushed after 240
Redistributing: rip
Default version control: send version 2, receive version 2
Interface Send Recv Triggered RIP Key-chain
FastEthernet0/0 2 2
FastEthernet0/1 2 2
Automatic network summarization is not in effect
Maximum path: 4
Routing for Networks:
5.0.0.0
136.1.0.0
Routing Information Sources:
Gateway Distance Last Update
136.1.45.4 120 00:00:08
Distance: (default is 120)

R5#ping 136.1.45.255

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.1.45.255, timeout is 2 seconds:

Reply to request 0 from 136.1.45.4, 8 ms
Reply to request 1 from 136.1.45.4, 32 ms
Reply to request 2 from 136.1.45.4, 16 ms
Reply to request 3 from 136.1.45.4, 4 ms
Reply to request 4 from 136.1.45.4, 24 ms
R5#

R4
show ip route
show ip route vrf Vrf1
show ip protocols
show mpls interface
show mpls ldp neighbor
show mpls forwarding-table
show ip bgp summary
show ip bgp vpnv4 all
ping vrf Vrf1 136.1.45.255
ping 136.1.34.255

R4#show ip route

Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

136.1.0.0/24 is subnetted, 2 subnets
O 136.1.23.0 [110/65] via 136.1.34.3, 00:04:28, FastEthernet0/0
C 136.1.34.0 is directly connected, FastEthernet0/0
150.1.0.0/32 is subnetted, 2 subnets
C 150.1.4.4 is directly connected, Loopback0
O 150.1.2.2 [110/66] via 136.1.34.3, 00:04:28, FastEthernet0/0
R4#show ip route vrf Vrf1

Routing Table: Vrf1
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

136.1.0.0/24 is subnetted, 2 subnets
B 136.1.12.0 [200/0] via 150.1.2.2, 00:04:23
C 136.1.45.0 is directly connected, FastEthernet0/1
1.0.0.0/24 is subnetted, 1 subnets
B 1.1.1.0 [200/2] via 150.1.2.2, 00:04:23
5.0.0.0/24 is subnetted, 1 subnets
R 5.5.5.0 [120/1] via 136.1.45.5, 00:00:01, FastEthernet0/1
R4#show ip protocols
Routing Protocol is "ospf 234"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 150.1.4.4
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
Routing on Interfaces Configured Explicitly (Area 0):
FastEthernet0/0
Loopback0
Reference bandwidth unit is 100 mbps
Routing Information Sources:
Gateway Distance Last Update
150.1.2.2 110 00:04:28
150.1.3.3 110 00:04:28
Distance: (default is 110)

Routing Protocol is "rip"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Sending updates every 30 seconds, next due in 0 seconds
Invalid after 180 seconds, hold down 180, flushed after 240
Redistributing: rip
Default version control: send version 1, receive any version
Automatic network summarization is in effect
Maximum path: 4
Routing for Networks:
Routing Information Sources:
Gateway Distance Last Update
Distance: (default is 120)

Routing Protocol is "bgp 24"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
IGP synchronization is disabled
Automatic route summarization is disabled
Neighbor(s):
Address FiltIn FiltOut DistIn DistOut Weight RouteMap
150.1.2.2
Maximum path: 1
Routing Information Sources:
Gateway Distance Last Update
Distance: external 20 internal 200 local 200

R4#show mpls interface
Interface IP Tunnel Operational
FastEthernet0/0 Yes (ldp) No Yes
R4#show mpls ldp neighbor

R4#show mpls forwarding-table
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
17 Untagged 150.1.2.2/32 0 Fa0/0 136.1.34.3
18 Untagged 136.1.23.0/24 0 Fa0/0 136.1.34.3
19 Untagged 5.5.5.0/24[V] 19950 Fa0/1 136.1.45.5
20 Aggregate 136.1.45.0/24[V] 1684
R4#show ip bgp summary
BGP router identifier 150.1.4.4, local AS number 24
BGP table version is 1, main routing table version 1

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
150.1.2.2 4 24 148 149 1 0 0 02:08:57 0
R4#show ip bgp vpnv4 all
BGP table version is 22, local router ID is 150.1.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf Vrf1)
*>i1.1.1.0/24 150.1.2.2 2 100 0 ?
*> 5.5.5.0/24 136.1.45.5 1 32768 ?
*>i136.1.12.0/24 150.1.2.2 0 100 0 ?
*> 136.1.45.0/24 0.0.0.0 0 32768 ?
R4#ping vrf Vrf1 136.1.45.255

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.1.45.255, timeout is 2 seconds:

Reply to request 0 from 136.1.45.5, 4 ms
Reply to request 1 from 136.1.45.5, 28 ms
Reply to request 2 from 136.1.45.5, 16 ms
Reply to request 3 from 136.1.45.5, 32 ms
Reply to request 4 from 136.1.45.5, 8 ms
R4#ping 136.1.34.255

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.1.34.255, timeout is 2 seconds:

Reply to request 0 from 136.1.34.3, 8 ms
Reply to request 1 from 136.1.34.3, 4 ms
Reply to request 2 from 136.1.34.3, 16 ms
Reply to request 3 from 136.1.34.3, 8 ms
Reply to request 4 from 136.1.34.3, 4 ms
R4#

R3
show ip route
show ip protocols
show mpls interface
show mpls ldp neighbor
show mpls forwarding-table

R3#show ip route

Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

136.1.0.0/24 is subnetted, 2 subnets
C 136.1.23.0 is directly connected, Serial0/0.23
C 136.1.34.0 is directly connected, FastEthernet0/0
150.1.0.0/32 is subnetted, 3 subnets
O 150.1.4.4 [110/2] via 136.1.34.4, 00:05:16, FastEthernet0/0
C 150.1.3.3 is directly connected, Loopback0
O 150.1.2.2 [110/65] via 136.1.23.2, 00:05:26, Serial0/0.23
R3#show ip protocols
Routing Protocol is "ospf 234"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 150.1.3.3
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
Routing on Interfaces Configured Explicitly (Area 0):
FastEthernet0/0
Serial0/0.23
Reference bandwidth unit is 100 mbps
Routing Information Sources:
Gateway Distance Last Update
150.1.4.4 110 00:05:16
150.1.2.2 110 00:05:26
Distance: (default is 110)

R3#show mpls interface
Interface IP Tunnel Operational
FastEthernet0/0 Yes (ldp) No Yes
Serial0/0.23 Yes (ldp) No Yes
R3#show mpls ldp neighbor

R3#show mpls forwarding-table
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
16 Untagged 150.1.2.2/32 0 Se0/0.23 point2point
17 Untagged 150.1.4.4/32 0 Fa0/0 136.1.34.4

R2
show ip route
show ip route vrf Vrf1
show ip protocols
show mpls interface
show mpls ldp neighbor
show mpls forwarding-table
show ip bgp summary
show ip bgp vpnv4 all
ping 136.1.23.255
ping vrf Vrf1 136.1.12.255

R2#show ip route

Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

136.1.0.0/24 is subnetted, 2 subnets
C 136.1.23.0 is directly connected, Serial0/0.23
O 136.1.34.0 [110/65] via 136.1.23.3, 00:05:41, Serial0/0.23
150.1.0.0/32 is subnetted, 2 subnets
O 150.1.4.4 [110/66] via 136.1.23.3, 00:05:41, Serial0/0.23
C 150.1.2.2 is directly connected, Loopback0
R2#show ip route vrf Vrf1

Gateway of last resort is not set

136.1.0.0/24 is subnetted, 2 subnets
C 136.1.12.0 is directly connected, FastEthernet0/0
B 136.1.45.0 [200/0] via 150.1.4.4, 00:05:18
1.0.0.0/24 is subnetted, 1 subnets
O 1.1.1.0 [110/2] via 136.1.12.1, 02:18:47, FastEthernet0/0
5.0.0.0/24 is subnetted, 1 subnets
B 5.5.5.0 [200/1] via 150.1.4.4, 00:05:18
R2#show ip protocols
Routing Protocol is "ospf 234"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 150.1.2.2
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
Routing on Interfaces Configured Explicitly (Area 0):
Serial0/0.23
Loopback0
Reference bandwidth unit is 100 mbps
Routing Information Sources:
Gateway Distance Last Update
150.1.4.4 110 00:05:41
150.1.3.3 110 00:05:51
Distance: (default is 110)

Routing Protocol is "bgp 24"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
IGP synchronization is disabled
Automatic route summarization is disabled
Neighbor(s):
Address FiltIn FiltOut DistIn DistOut Weight RouteMap
150.1.4.4
Maximum path: 1
Routing Information Sources:
Gateway Distance Last Update
Distance: external 20 internal 200 local 200

R2#show mpls interface
Interface IP Tunnel Operational
Serial0/0.23 Yes (ldp) No Yes
R2#show mpls ldp neighbor

R2#show mpls forwarding-table
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
17 Untagged 136.1.34.0/24 0 Se0/0.23 point2point
18 Untagged 150.1.4.4/32 0 Se0/0.23 point2point
21 Aggregate 136.1.12.0/24[V] 1040
22 Untagged 1.1.1.0/24[V] 18240 Fa0/0 136.1.12.1
R2#show ip bgp summary
BGP router identifier 150.1.2.2, local AS number 24
BGP table version is 1, main routing table version 1

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
150.1.4.4 4 24 151 150 1 0 0 02:10:15 0
R2#show ip bgp vpnv4 all
BGP table version is 13, local router ID is 150.1.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf Vrf1)
*> 1.1.1.0/24 136.1.12.1 2 32768 ?
*>i5.5.5.0/24 150.1.4.4 1 100 0 ?
*> 136.1.12.0/24 0.0.0.0 0 32768 ?
*>i136.1.45.0/24 150.1.4.4 0 100 0 ?
R2#ping 136.1.23.255

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.1.23.255, timeout is 2 seconds:

Reply to request 0 from 136.1.23.3, 12 ms
Reply to request 1 from 136.1.23.3, 16 ms
Reply to request 2 from 136.1.23.3, 32 ms
Reply to request 3 from 136.1.23.3, 28 ms
Reply to request 4 from 136.1.23.3, 28 ms
R2#ping vrf Vrf1 136.1.12.255

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.1.12.255, timeout is 2 seconds:

Reply to request 0 from 136.1.12.1, 4 ms
Reply to request 1 from 136.1.12.1, 28 ms
Reply to request 2 from 136.1.12.1, 16 ms
Reply to request 3 from 136.1.12.1, 28 ms
Reply to request 4 from 136.1.12.1, 12 ms
R2#

R1
show ip int brief | ex una
show ip route
show ip protocols
ping 136.1.12.255

R1#show ip int brief | ex una

Interface                  IP-Address      OK? Method Status                Protocol

FastEthernet0/0            136.1.12.1      YES NVRAM  up                    up

FastEthernet0/1            1.1.1.1         YES manual up                    up

Loopback0                  150.1.1.1       YES NVRAM  up                    up

R1#show ip route

Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

136.1.0.0/24 is subnetted, 2 subnets
C 136.1.12.0 is directly connected, FastEthernet0/0
O E2 136.1.45.0 [110/1] via 136.1.12.2, 00:06:20, FastEthernet0/0
1.0.0.0/24 is subnetted, 1 subnets
C 1.1.1.0 is directly connected, FastEthernet0/1
5.0.0.0/24 is subnetted, 1 subnets
O E2 5.5.5.0 [110/1] via 136.1.12.2, 00:06:20, FastEthernet0/0
150.1.0.0/24 is subnetted, 1 subnets
C 150.1.1.0 is directly connected, Loopback0
R1#show ip protocols
Routing Protocol is "ospf 12"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 150.1.1.1
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
1.1.1.0 0.0.0.255 area 0
136.1.12.1 0.0.0.0 area 0
Reference bandwidth unit is 100 mbps
Routing Information Sources:
Gateway Distance Last Update
136.1.12.2 110 00:06:20
Distance: (default is 110)

R1#ping 136.1.12.255

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.1.12.255, timeout is 2 seconds:

Reply to request 0 from 136.1.12.2, 8 ms
Reply to request 1 from 136.1.12.2, 20 ms
Reply to request 2 from 136.1.12.2, 12 ms
Reply to request 3 from 136.1.12.2, 24 ms
Reply to request 4 from 136.1.12.2, 8 ms
R1#

What do you see as the problem, and what may be done to correct it?

Good luck!