In the final stages of a network upgrade in a K12 school.
All new switches in place, with an IRF stack of 5700 at the center. These 5700s have BAGGs to individual HP 2920s that are placed around the school campus.
The previous network was using a 5412zl for routing, and our goal is to retire this switch (single point of failure compared with IRF stack )
We have 27 VLAN interfaces, so we are hopefully not hitting the limit of 32 on the 5700s.
We created two VRFs
#
ip vpn-instance vpn10001
route-distinguisher 10001:1
vpn-target 65001:10001 import-extcommunity
vpn-target 65001:10001 export-extcommunity
#
#
ip vpn-instance vpn10002
route-distinguisher 10002:1
vpn-target 65001:10002 import-extcommunity
vpn-target 65001:10002 export-extcommunity
#
The second VRF is for guest internet access, and it is the only VLAN interface bound to the VRF
interface Vlan-interface190
description VLAN0190
ip binding vpn-instance vpn10002
ip address 10.2.0.1 255.255.252.0
The first VRF is for the rest of the production network traffic, and each of the 26 remaining VLAN interfaces was bound to it using commands like:
interface Vlan-interface185
description VLAN0185
ip binding vpn-instance vpn10001
ip address 10.1.185.1 255.255.255.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
default routes were added to each VRF
ip route-static vpn-instance vpn10001 0.0.0.0 0 10.1.99.2
ip route-static vpn-instance vpn10002 0.0.0.0 0 10.2.0.2
At this point we literally went to every building, got on each individual VLAN with multiple clients, and verified routing / Internet access was working and it was.
Then came the following morning...
Symptoms included some clients working fine, while others could not ping their gateway IP. We were able to find two clients on the same VLAN on the same 2920 and confirm that one could ping the GW, while the other could not, yet they could ping each other. At first we looked at spanning tree ( all switches configured for MSTP ) and could find no ports in abnormal conditions or any signs of any recent STP events.
We ended up removing all the VLAN Interface entries and moved routing back to the 5412zl, and all is working now.
I did NOT get a copy of the config running on the 5700s at the time of the incident, but what I do have is the current running config, and a text file that i created for the purpose of "moving" the routing functionality to the 5700 switches.
Note -> we did leave the smaller VRF on the 5700 and it is working fine for guest internet ( we never had any issues with this VRF during the incident ).
Anyone have any ideas how to troubleshoot this scenario?
The procedure used to move routing was simple. First we downed the interface to the 5412zl, then i pasted the following into the 5700 config:
#
ip vpn-instance vpn10001
route-distinguisher 10001:1
vpn-target 65001:10001 import-extcommunity
vpn-target 65001:10001 export-extcommunity
#
#
ip vpn-instance vpn10002
route-distinguisher 10002:1
vpn-target 65001:10002 import-extcommunity
vpn-target 65001:10002 export-extcommunity
#
interface Vlan-interface40
description VLAN0040
ip binding vpn-instance vpn10001
ip address 10.1.40.1 255.255.255.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface099
description VLAN0099
ip binding vpn-instance vpn10001
ip address 10.1.99.1 255.255.255.0
interface Vlan-interface100
description VLAN0100
ip binding vpn-instance vpn10001
ip address 10.1.100.1 255.255.255.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface111
description VLAN0111
ip binding vpn-instance vpn10001
ip address 10.1.111.1 255.255.255.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface110
description VLAN0110
ip binding vpn-instance vpn10001
ip address 10.1.110.1 255.255.255.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface112
description VLAN0112
ip binding vpn-instance vpn10001
ip address 10.1.112.1 255.255.255.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface115
description VLAN0115
ip binding vpn-instance vpn10001
ip address 10.1.115.1 255.255.255.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface120
description VLAN0120
ip binding vpn-instance vpn10001
ip address 10.1.120.1 255.255.252.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface124
description VLAN0124
ip binding vpn-instance vpn10001
ip address 10.1.124.1 255.255.252.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface129
description VLAN0129
ip binding vpn-instance vpn10001
ip address 10.1.129.1 255.255.255.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface132
description VLAN0132
ip binding vpn-instance vpn10001
ip address 10.1.132.1 255.255.252.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface136
description VLAN0136
ip binding vpn-instance vpn10001
ip address 10.1.136.1 255.255.252.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface140
description VLAN0140
ip binding vpn-instance vpn10001
ip address 10.1.140.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface150
description VLAN0150
ip binding vpn-instance vpn10001
ip address 10.1.150.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface152
description VLAN0152
ip binding vpn-instance vpn10001
ip address 10.1.152.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface160
description VLAN0160
ip binding vpn-instance vpn10001
ip address 10.1.160.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface162
description VLAN0162
ip binding vpn-instance vpn10001
ip address 10.1.162.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface170
description VLAN0170
ip binding vpn-instance vpn10001
ip address 10.1.170.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface172
description VLAN0172
ip binding vpn-instance vpn10001
ip address 10.1.172.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface180
description VLAN0180
ip binding vpn-instance vpn10001
ip address 10.1.180.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface182
description VLAN0182
ip binding vpn-instance vpn10001
ip address 10.1.182.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface185
description VLAN0185
ip binding vpn-instance vpn10001
ip address 10.1.185.1 255.255.255.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface190
description VLAN0190
ip binding vpn-instance vpn10002
ip address 10.2.0.1 255.255.252.0
interface Vlan-interface200
description VLAN0200
ip binding vpn-instance vpn10001
ip address 10.1.200.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface202
description VLAN0202
ip binding vpn-instance vpn10001
ip address 10.1.202.1 255.255.254.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface204
description VLAN0204
ip binding vpn-instance vpn10001
ip address 10.1.204.1 255.255.252.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
interface Vlan-interface208
description VLAN0208
ip binding vpn-instance vpn10001
ip address 10.1.208.1 255.255.252.0
dhcp select relay
dhcp relay server-address 10.1.100.32
dhcp relay server-address 10.1.100.33
#
bgp 65001
router-id 192.168.21.2
#
address-family ipv4 unicast
import-route direct
#
ip vpn-instance vpn10001
#
address-family ipv4 unicast
import-route direct
#
ip vpn-instance vpn10002
#
address-family ipv4 unicast
import-route direct
ip route-static vpn-instance vpn10001 0.0.0.0 0 10.1.99.2
ip route-static vpn-instance vpn10002 0.0.0.0 0 10.2.0.2
ntp-service source Vlan-interface115