Controllerless Networks

I am working on an issue at a customer site that is quite perplexing and I have not been unable to uncover the root cause; 

Random IAPs (115s) disappear from the virtual controller running on campus. Airwave also shows that they are down which means they have not been separated from the VC and formed their own cluster. However, I am able to ping the IP that they were assigned and I can SSH to the IP and console access the "down" IAP. 

The weirdness starts in that I cannot ping the VC IP address or the IP of the AP that is functioning as the VC from the console of the "down" IAP. However, I can ping other IAPs that are showing as "up" in the VC and I can ping the subnet gateway that the VC and the "down" IAP share from the console of the "down" IAP.

I also am unable to ping the "down" IAP IP address from the VC even though I can ping it from my desktop whether I am on a different or same VLAN as the VC and the "down" IAP. From the VC console I can ping the other IAPs that the "down" IAP can ping as well as the shared subnet gateway.

All IAPs are on a dedicated VLAN (100) across campus and there is a total of about 80 IAPs on campus. There are multiple buildings on campus and VLANs span buildings. All switching is HP Procurve, dedicated VLAN specifically for IAPs and wireless users have their own VLANs. IAP models are 115s mostly with about ten 225s mixed into high usage areas. Current code running on IAPs is 6.4.2.0-4.1.1.0. 

Interesting note is that we are seeing this issue on the 115s and not once has it occurred on the 225s. I have looked at the client load on all IAPs and not one exceeds more than 30 users. In fact the ones that have been experiencing the issue have very low users counts >10. 

If we reboot the IAP it comes back but experiences the issue again at some random later time. We have tried factory resetting the IAPs with the issue and all seemed fine for them over the weekend but at 9am this morning it began happening again. 

The issue is not on every IAP. We are seeing this on about 20 IAPs across campus. 

The VC AP has stayed consistent and the VC role has not roamed across IAPs. 

Anyone else seeing anything similar? I contacted another of my customers with a similar deployment and they noted something similar with same code version and IAP model but did not do enough troubleshooting to state that it was exactly the same. They rebooted the IAPs and all was well. 

did you ever get a fix for this

I have 6 iap 205 and they show the extact same kind of behaviour

Are they running version 6.4.2.0-4.1.1.0?

We reverted to an older code and everything stabilized. 

You also might want to try updating to newer code and see if the issue disappears. 

we tried it on the original code that was installed when we received them and the 4.1.1.1 code

both seemed to work for a day or so then intermittent network connectivity between the iap205's

from my pc on another vlan I never lose connection to the ap's

I currently have a ticket open and will share our findings in case it can help some one else

thanx

Check to make sure that the VC ip is not being handed out in dhcp.  When the issue happens, try to get a wireshark on that vlan, and you will soon see if there is a duplicate ip.

the vc ip is excluded from the dhcp scope

and i only have 5 iap205's

but I will try out the wireshark idea maybe something weird is happening on that vlan

I am seeing some "wonkiness" with the 6.4.2.0 code across multiple AP models and customer implementations. For now I am going to stay with 6.3.1.x since I have not seen any issues with this code at any of several dozen customer implementations we support. 

The issues I have seen are; 

IAPs that disappear from VC but still can be accessed via WebUI and SSH as described in my original post. 

IAPs that go unreponsive and a reboot resolves the issue only to randomly occur again

Virtual IP roams to new IAP when original master fails but IP becomes unreachable. ARP table from original master IAP switch still shows virtual IP bound to switch port with original not current master IAP connected to it. 

IAP boots, gets IP address from network and then loses it only to get an autoconfig address. 

We implement with best practices; dedicated VLAN just for the IAPs, small broadcast domains, exclusion of virtual IP from DHCP scope, dedicated wireless user VLANs, etc... We have many customer sites of differing complexities running older non 6.4 code and they are stable and reliable. 

These issues are being seen on IAP 115s, 205s, and 225s. The customers are a mix of Cisco, HP and Brocade switch environments. 

unfotunately all my IAP's are 205 which only work on the 6.4.2.0 code

I think they are working on a new version as I have seen 6.4.2.0 - 4.1.2 on the support site

so I will wait for the support team to let me know what the next steps are