Controllerless Networks

We are having fairly major issues around our campuses. It all started with the plan to replace one of our 105s with a 225 at 4 of our 5 networks. 3 of these 225s were to go into a digital arts classroom that could benefit from the extra bandwidth and the 4th one in the IT area. As of now, all but one campus are for the most part settled.

I first tested one at the network 10.1.x.x and realized these would not just auto config out of the box. It turned out to be the firmware they were shipped with. All our controllers are on 4.0.0.4. I isolated the Aps on a separate VLAN at this location and attached to them via 'instant' and upgraded the firmware. Once I did this, I was able to wipe the config and plug in it at this location and it showed up no problem. I only plugged in the first port as we had to wait for the cabler to run a second drop to that location. This seemed to go just fine. Later when the cabler ran the 2nd cable to the 225, it caused the controller to disappear and it did not reappear until I rebooted all switches (All Aruba on this network for Aps)

The 2nd one installed was at the network 10.5.x.x. I was not there when it was plugged in, but I was immediately called and told wireless was down. I could not reach the assigned IP for the virtual controller. Rebooting all the Aruba switches ended up fixing it, but  we still have some on Cisco 3750 switches that I had to shutdown and restart to eventually see the virtual controller again. This one SEEMS settled now and I have not touched it since. Although in the process of these issues, we seemed to have lost 3 Aps. 2 on Cisco switches and 1 on an Aruba switch. They no longer are showing up in the controller. Because we are so dense in the number of Aps we have, this hasn't been a major problem yet.

The 3rd one installed was at the network 10.2.x.x. This one went smooth and I have no heard any issues at all. I speculate this could be because we have all Aruba switches providing power and data to 100% of the Aps. This is also true at the network 10.1.x.x but not at 10.5.x.x, 10.4.x.x (not getting a 225 anyway) or 10.3.x.x.

The 4th one was at the network 10.3.x.x. This one also seemed to go smooth. However, the following day from installing the 225, we were completely down. Pretty much the same thing that happened at 10.5.x.x except delayed. This is our biggest campus and I have been left with around 20 Aps down. After fixing 10.3.x.x up best I could, the symptoms occurred again. I had a technician pull the 225 at this point and rebooted everything, again, only to have the same 20 Aps not come up still. I have reset Aps on both Cisco and Aruba switches in an effort to get any of these back up but have had no luck. Both radio lights stay amber. I have reset them via the button and console cable. I am at a total loss at this point. I am unsure how to proceed.  I have a TAC open but I am curious what the community has to say.  This is 5 separate networks we run.  4 of them we added 225s and 3 of those we have had issues (so far).

Please continue to work with TAC for troubleshooting this issue.  Mixing AP models should not be a problem.  You just need to make sure that the SW version match on the new APs **if the version running on the VC does not exist up in the cloud**.  In that case you must manually upgrade first.  Also, you cannot mix APs with different Regulatory Domains in the same cliuster.   If in the US they must state -US in the part number (IAP-105-US).  If they are elsewhere they must be the "Rest of World" version -RW (IAP-225-RW).

You mentioned using both ports on the AP-225.  Did you create a LAG group, static or with LACP, on your switches?  If not you will create a loop.  STP blocking?

When the network went down, was the IAP-225 cabled up to the second ethernet port?

There is a defect which has been recorded and will be fixed in the next release which causes cluster failure when using ethernet 1

100205 - Connect IAP’s eth1 to switch by mistake may cause VC

               [MasterElection]: prevent master election from e1 by damage the cluster.

                  Stop sending master beacon, hierarchial beacon to downlink port if uplink is not UP;

                  Ignore any beacon from IAP, if that IAP is not in my allowed-ap-list.

Marcus.  I spent almost 6 hours on the phone with support yesterday.  Unfortunately they seemed confused as well.  I think what Scott mentions is what may have happened.  My question though is why.  Probably because I dont even know what a LAG group is.  So I certainly did not set one up.  lol  I mentioned this to support as well but since I had pulled the 225 at this point, I guess he was just more concerned with fixing the cluster first.  Which they were unable to before I got crabby (yes I did say 6 hours on the phone) and demanded we pick up in the morning.  

Last night I ended up fixing 10.3.x.x by shutting EVERY AP off and bringing them up one IDF at a time.  After the first one booted I was unable to connect to what I had set the VC to be, but found what IP was grabbed by the AP in DHCP and connected to that.  'Instant' SSID only.  I had backups of the config but it's easy enough to setup 5 SSIDs and tweak a few settings, so I just reset up the controller.  Every other AP in the cluster connected to the new controller no problem.

I also had issues at the 10.2.x.x at some point where it was not responding to the VC ip I had previously set.  It was also crumbling.  I quickly shut down data and poe to both ports on all 3 remaining 225s that were up.  THIS caused 10.1.x.x and 10.5.x.x to lose their controllers.  I rebooted all APs one IDF at a time at each campus and everything was happy again.  I did not have to reset all those up at least.

Now I have 5 very happy WLANs but without 225s.  Scott, your response indicates to me that there is an issue with this device and I should hold off using it at all, yet.  Is this the case?  Or should I research and look into setting up a LAG group Marcus?  Can you assist me?  We are using 2500 switches where all 4 225s will be.

If nothing else, I would like to plug eth0 in with the 225s to get the coverage back in those classrooms (although they get enough bleedover from adjacent classrooms),

Thank you both for the insight.  Any advise on how to proceed?

Brian

PS.  I guess I should clarify because I didnt outright say it yet.  Yes.  I had both ports plugged in on the IAP225s to the same switch per AP.  In all cases it is an Aruba S2500-24P

The bug Scott mentioned is scheduled to be fixed in the 4.1 release in a couple of weeks.  In the meantime I would suggest bringing the AP-225s up with a single Ethernet connection.  Be sure you are connected to Eth0 or you will encounter the issue Scott mentioned.  Another thing to try is to bring the AP-225 up on Eth0.  Once online then connect Eth1 -- do this ONLY after you have configured a LAG.

LAG = Link Aggregation Group.  Configuring a LAG on your switch is an absolute requirement if using both ports.  This takes two physical ports and binds them, logically, to act as 2Gig port.  There is "Static LAG" and LAG with LACP (Link Aggregation Controll Protocol) that dynamically manages the links in the LAG.

Now, do you really need 2Gig to an AP-225?  That depends on your usage.  It is possible that an AP-225 with a single link to the switch could over-subscribe that port but it is unlikely..........you would need a 3 stream .11ac client pushing line rate plus a 3stream .11n 2.4Ghz client pushing line rate.   Now, when Wave 2 of 802.11c arrives that will introduce new HW with new features (mult-user MIMO) that will be able to take advantage of two Ethernet links.

http://en.wikipedia.org/wiki/Link_aggregation

http://en.wikipedia.org/wiki/Link_Aggregation_Control_Protocol#Link_Aggregation_Control_Protocol

Not sure if this is in the current release or coming but this is what came back to me from TAC / Support

Plugging IAP in Eth1 will result in the setup being Hierarchy deployment.

All the Aps connected to the VLAN in eth1 will become slaves including the current VC master. This is a feature called Hierarchy deployment.

i think this is a new feature which isn't yet documented in the user guide.

As to the dual port uplinks, you definitely need more configuraton to get this going so maybe just start with one. Realistically 2 Gig uplinks won't be needed until you have a full 802.11ac environment (clients and AP's) and even then you'd be lucky to break 1 Gig. Wave 2 will be a different story.

I have the 225s all on their respective WLANs using eth0 only, for the time being.  Thank you for all the help and advice.  It is much appreciated.