Controllerless Networks

Hi,

I recently ran into an issue with a new IAP 225 cluster (ArubaOS (MODEL: 225), Version 6.3.1.1-4.0.0.1 (41049)).

Before i could get to configuring it, somone must have logged in and done something. The IAP's seem to be unable to keep their IP, it gets an IP during boot, but when its "up" it does not broadcast any SSID, it decides to show an IPv6 address when viewed on the switch via LLDP, or a default ip, ex 169.x.x.x.

After that cycle I am unable to contact the AP via webinterface again.

If i add a new IAP to the same subnet, the AP comes up and gets another master and then reboots and goes into same cycle.

I tried shutting down all IAP and setting up a configured IAP, but when i enable power they never join the new cluster.

I tried regaining control by adding DHCP option 43 to get it redirected to Airwave, but it does not show up.

AP's are already mounted in ceilings etc. so I am unable to gain console access easily and would love to get some suggestions that would not require me to go fetch a ladder and manually reset a bunch of AP's :)

So, how do one regain control of a cluster you can't get on the webinterface, nor easily manually reset them? :)

Regards

Kevin

Kevin,

There are a myriad of reasons why you would be encountering your issue.  Your best bet is to open a case with TAC.   OR get a ladder and take that AP down from the ceiling, because to get diagnostics from it or to understand what is happening, you will have to plugin the console anyways.

If you have all your AP's on Poe switches that are managable, you could power them all down, then connect up a clean / factory reset AP and then try powering them all back on. With a bit of luck they may all join the new master and overwrite their configuration.

I've done this before but have seen recent reports suggesting you need to factory reset an IAP when moving between clusters so this may not work.

Scott

@Scott,

Thats exactly what I'm trying to do, and everywhere I look, it says regardless of current configuration, if theres an existing cluster, the IAP will join that cluster.

This is just not the case. 

I can get a cluster running with ex. 2 AP's, then I enable PoE on one of the troublesome AP's and I loose connectivity to the existing cluster.

Is there any way to configure the cluster to be "boss-cluster" ? Did they change the way to merge clusters in 4.0.0.x ?

I enabled Preferred Master on the good cluster and this seems to allow it to stay alive, but I never get any joins to that cluster.

@Colin

Any commands to get closer to why the other cluster gets to be "boss-cluster" ?

Regards

Kevin

We saw an issue like this.

In our case a cluster of 10-12 iAP would be getting along fine, then one of them would stop broadcsting the SSIDs and gradually all of them but the cluster-master would eventually stop broadcasting as well. Generally we couldn't get the non-broadcasting units to let us authenticate on SSH or in the GUI.

In out syslog server we saw the sequence would start with the first one to stop broadcasting apparently loosing "sight" of the master and beginning to claim to be master. Eventually the others would beging to look to the first as master and the lonely master of the original cluster would be left hollering for followers.

We were able to get back control by shutting off power to all iAP, then bring one up and upgrade it to 6.3.1.2-4.0.0.4_42384 then bring up one more at a time until all were on newer code. We haven't seen the problem since.

Hi Kevin,

When i did this it was back in the first few releases of instant. Obviously something has changed somewhere along the line. I just checked the instant user guide for 4.x and there is a line stating "Moving an IAP from one cluster to another requires a factory reset of the IAP."

So it sounds like the best option for you would be to take one of the "fault" IAP's, console in and reset password if needed, then perform whatever changes are needed to get things back to manageable. Maybe a firmware upgrade?

Scott

We had a tech retrieve two of the iAP which were affected from one of our stores and in the lab brought them up and found that they still remembered the cluster they had been in, so there was no problem getting them to join the upgraded master iAP, it was the same cluster as before, just newer code.

@msabin

Thank you for the suggestions, I will try to see if i can get them up solo and get them upgraded.

@Scott

When you say console in and change password, is that doable if current password is unknown ? Would that not pose a security risk if anyone could plug in to your IAP cluster and change the password ?

Regards 

Kevin

We took a 20 minute window and brought all iAP in a cluster down, brought one up, upgraded it then brought the others on one-at-a-time and they detected the upgraded master iAP and self-upgrade and join the party.

Got 12 done in 20 minutes. (well maybe 25)

On the console password point, yes, anyone with console access could do your some harm -- but generally our iAP are 30 feet up, or in office area ceilings where a BADGUY would be seen and questioned.

So finally got the cluster up and running, the solution was already on the forums. I had to use the Eth0 port of the AP's.

The little twist that threw me off course was that you cannot have even one AP connected with Eth1, that will blow up your cluster.

When all AP's was reconnected to Eth0, everything was fine.

@msabin

We tried bringing all AP's down, cable a new AP to Eth0 and upgrade the firmware to the latest, but when we turned on an AP connected to Eth1, it didnt upgrade, instead it pulled the existing cluster down again.

Thank you all for your help and input.

/Kevin

Thanks for the tip on eth0 vs eth1 -- wouldn't have called that one. Then again, we haven't used any iAP with an eth1 yet.