Wired Intelligent Edge

I'm having a issue with one of our sites running an IAP cluster of 78 AP225s, I'll try to describe the issue the best I can.

The school consists of a Procurve 5408zl switch as the core, and aruba MAS s2500 switches for all of the access points and all of the devices in different buildings have s2500s as well.  From my office I can ping the HP switch, and the Aruba S2500 with no packet loss, and approx 1ms ping times.  From my office when I ping the VIP I have high packet loss, and long ping times.  I then SSH into the s2500 and ping the VIP with the same high packet loss.  The part making it a pain to track down is that things will work great, low ping times to everything and happy users.  Then at a random time of the mornings I will run into the same problem, then 20-30 mins later the problem goes away and may or may not return the same day... fun.

I've been checking the logs on the HP swtich and s2500s, but nothing is jumping out at me.  Any thoughts on what I can check?

The s2500s are running 7.4.1.1

The IAP 225s are runing 6.4.3.1-4.2.0.1_51844.

Hi, 

If the problem occurs for only 20-30 mins in the morning, is there anything that happens during this time? 

- Is this the time at which most of the users login / connect to wifi? 

- Is there much client roaming during this time? 

- Are the users users stationary when the issue is not seen? 

- Anything else that you think that happens during this time interval? 

And, how many IAPs are in the cluster and what is the max number of wifi clients on this cluster? 

Thanks, 

Rajaguru Vincent 

Thanks for getting back to me.  The time that I start seeing this varies daily.  Some days I see it at 8:30 am, then it all works well until another random time the next day... I get reports of high packet loss form a solarwinds product as well, sometimes it will report problems at midnight when no one is on campus at all.

- The times seem to vary from day to day making it a little tricky to test for.

- Their should not be any more users or roaming during this time, school starts at 9:00. we saw the problem today around 9:50 and it's been good for several hours since... 

- Yes, the users are generally in a class working at their desks.  

- Nothing I can think of changes during this time

This cluster contains 78 APs, currently (working fine) there are 350 users.  Daily we see approximatly 500-600 concurrent connections.

When did you start noticing the issue? Were there any changes in the config or in the network that could have triggered the issue like, 

- After adding extra IAPs to the cluster? 

- After upgrading the IAP firmware version? 

- Any configs changes in th IAP or Switch? 

 And, is Client Match enabled in this cluster? 

Thanks, 

Rajaguru Vincent 

I've been trying to track down the issue for sometime.  Since seeing the problems I have enables STP in airwave on the switches, enabled rogue AP protection with a 300 second time out, and updated the firmare on them at least one.  

The IAPs are all configured the same way that I've got 16 other IAP clusters that are all about the same size (fewer clents though) and the other sites are working fine.  I have also updated the firmware on the IAPs at least once during troubleshooting.

Client Match is enabled right now.

Thanks again!

Dennis

Given the users are happy and everything else is ok, the first thing that springs to mind is a duplicate ip for the VC-ip.

The APs are pulling their IP addresses from DHCP.  I did set the VC IP as a static IP on the cluster, is there any log I could check to see if it is somehow a duplicate?

Thanks for any insight :).

Dennis

Did you exclude the VC IP from the DHCP scope? Do you see a DHCP lease for the VC IP in the DHCP server?

Yes, DHCP does not give any addresses out in that range.  I just changed the IP to another known free address and as soon as the IP change became active nothing started responding on the old address.

Could you try disabling Client Match and see if there is a difference.

Even though Client Match is related to RF, there is more broadcasts (IAP messages) on the wired side seen during Client Match triggers in some cases. There could be some network disruption during that time.

This is not expected when there are no users. However, you can give a try.

Thanks,
Rajaguru Vincent

I can give that a shot, thanks for the info.

I did not know that the wireless connections could impact the devices wired interface.  I have a feeling the problem may be related to the wired side... Right now the school has a flat 16-bit VLAN for all traffic (ick).  Our wired devices, eth0 on the AP225s and the wireless clients are on the same vlan.  I know this could cause problems with the wireless clients - however could this also impact the management interface?

If yes, I am thinking, and please correct me if I'm wrong, that I should setup a VLAN for only the APs wired interfaces, then a vlan that the IAP drops clients onto.  I have done this in some test enviornments but as best as I can tell I have to set the AP ports to a profile that I have to manually set to the correct VLAN - is it possible on the MAS switches when I plug an aruba device into them to detect that and stick them on the right vlan (I have done this with LLDP and VoIP devices on other switches).

Thanks for any input, hopefully I'm on the right track.

Dennis

ddevore,

Yes, you should have the access points on a different management VLAN than the clients that connect to them, especially in larger clusters.  Please see sriram's advice in the thread here;  http://community.arubanetworks.com/t5/Aruba-Instant-Cloud-Wi-Fi/Settings-for-New-Install-At-Middle-School/td-p/87962  Yes, having too much traffic can affect the management interface, especially wired broadcast traffic.

There is no way to automatically detect an IAP and set a VLAN, no.

You will probablyhave to:

- create a single additional vlan on your HP Layer 3 switch

- extend it to all of your MAS switches via trunks between switches and trunks on AP interfaces

- Put your wireless users on that new VLAN by changing the VLAN number in the SSID configuration

Since you have a /16 you might even consider creating an additional VLAN for AP management and extending that to your switches as well.  You would make that new  management vlan the untagged or default VLAN on those ports and then have the wireless client vlan be the tagged vlan on those ports.  In the long term, this would provide the greatest stability and performance for your network.

If you feel you cannot handle this redesign, please get some professional advice so that things are less painful.  You already have a lab setup, so you look like you are on your way.

Thanks for the info.  I have everything working fine in a test enviornment - I used some info from http://community.arubanetworks.com/t5/tkb/articleprintpage/tkb-id/WiredNetworks/article-id/159 to setup the aruba ap device group that seems to drop APs into the correct profile groups when it sees them connect.

I'll probably have to wait until the weekend to do any major network changes at the school site, would having the wired / wireless and management on the same VLAN cause that much loss to a wired port?  I'm just curious why that would happen with no norticible problems on wired comptuers or devices on the same vlan.

Thanks, have a nice day!

Dennis