01-11-2013 01:11 PM
We are having quite a weird issue happening on campus between our Master and Local controllers. I recently installed two new M3 controllers into an environment of one master and one other local. I created the ipsec tunnels between the master and the new locals and verified that they both have the correct key in place, however, I cannot ping the new locals from the master and vice versa, but all of the locals, including the existing local, can ping each other. They all reside on the same VLAN but different Cisco 6500 core switches. The weird thing is, if I setup a port channel on the master controller, then the locals and master can see each other and our VRRP configurations work fine but we then get issues with some clients not being able to route correctly. If I disable a port in the port channel on the master the clients return to normal but the local controllers can no longer communicate with the master if they are rebooted saying cannot heartbeat with the master. I should also mention that when one of the ports in the port channel is disabled on the master it takes over as the master for all the VRRP connections when it's supposed to be backup for some, and the locals also stay as master for their VRRP connections; the master can't seem to see the VRRP advertisements from the locals. Could the master be blocking the connection in some way? Or has anyone else seen an issue like this?
Solved! Go to Solution.
01-12-2013 12:59 AM
First thing I would check is the port configuration, what is the duplex and speed for the connection from and to controller.
- Check the network setup, gateway, L2/L3 setup on your network
- Check if master IP is defined on local, and local IP defined on master (or 0.0.0.0)
- Try remove the IPSec you mentioned and ping each other'
If you are able to ping from Master to Local then proceed to create IPSec tunnel.
01-14-2013 10:05 AM
I have verified the gateway and HSRP setup for that network, I even changed the priority on our main router for that vlan so one of the other routers would become the active router to make sure it wasn't something with our main 6500. These controllers all reside on the same subnet. I also tried removing the local IP information from the master and rebooted the local and they could not ping each other. I added the local IP and key back into the master, as well as verifying that the master IP is in the local with the key and rebooted the local and they still couldn't ping each other. It just seems to me like there's something odd going on with the master because all the locals can ping each other.
04-29-2013 12:40 PM
I just wanted to update this issue in case anyone else runs into this...The problem had to do with CDP noticing a Native VLAN Mismatch; I'll explain our setup and what was happening. We have one of our controllers running the VPN/RAPs and a remote site that has two 8 port Cisco switches running through a RAP5 which is what was causing the issue. The controller that runs the RAPs is connected to a Cisco 6500 and has a Native VLAN of 45. Our switches are all managed on VLAN 1 so because the remote switches were connected through the RAP, the 6500 was seeing a Native VLAN Mismatch. After researching what these log messages meant I found that CDP seeing a Native VLAN Mismatch causes STP to put that port into a port VLAN ID inconsistent state and won't forward traffic on the link. Once this problem was discovered we changed the remote switches to have their management IP's on VLAN 45 and this cleared up the issue. All controllers can ping each other and VRRP is working as expected.