11-30-2015 11:32 AM
I'm having a issue with one of our sites running an IAP cluster of 78 AP225s, I'll try to describe the issue the best I can.
The school consists of a Procurve 5408zl switch as the core, and aruba MAS s2500 switches for all of the access points and all of the devices in different buildings have s2500s as well. From my office I can ping the HP switch, and the Aruba S2500 with no packet loss, and approx 1ms ping times. From my office when I ping the VIP I have high packet loss, and long ping times. I then SSH into the s2500 and ping the VIP with the same high packet loss. The part making it a pain to track down is that things will work great, low ping times to everything and happy users. Then at a random time of the mornings I will run into the same problem, then 20-30 mins later the problem goes away and may or may not return the same day... fun.
I've been checking the logs on the HP swtich and s2500s, but nothing is jumping out at me. Any thoughts on what I can check?
The s2500s are running 18.104.22.168
The IAP 225s are runing 22.214.171.124-126.96.36.199_51844.
11-30-2015 11:48 AM
If the problem occurs for only 20-30 mins in the morning, is there anything that happens during this time?
- Is this the time at which most of the users login / connect to wifi?
- Is there much client roaming during this time?
- Are the users users stationary when the issue is not seen?
- Anything else that you think that happens during this time interval?
And, how many IAPs are in the cluster and what is the max number of wifi clients on this cluster?
11-30-2015 01:35 PM
Thanks for getting back to me. The time that I start seeing this varies daily. Some days I see it at 8:30 am, then it all works well until another random time the next day... I get reports of high packet loss form a solarwinds product as well, sometimes it will report problems at midnight when no one is on campus at all.
- The times seem to vary from day to day making it a little tricky to test for.
- Their should not be any more users or roaming during this time, school starts at 9:00. we saw the problem today around 9:50 and it's been good for several hours since...
- Yes, the users are generally in a class working at their desks.
- Nothing I can think of changes during this time
This cluster contains 78 APs, currently (working fine) there are 350 users. Daily we see approximatly 500-600 concurrent connections.
11-30-2015 02:53 PM
When did you start noticing the issue? Were there any changes in the config or in the network that could have triggered the issue like,
- After adding extra IAPs to the cluster?
- After upgrading the IAP firmware version?
- Any configs changes in th IAP or Switch?
And, is Client Match enabled in this cluster?
11-30-2015 04:19 PM
Given the users are happy and everything else is ok, the first thing that springs to mind is a duplicate ip for the VC-ip.
If my post is helpful please give kudos, or mark as solved if it answers your post.
ACCP, ACMP, ACMX #294
11-30-2015 04:45 PM
I've been trying to track down the issue for sometime. Since seeing the problems I have enables STP in airwave on the switches, enabled rogue AP protection with a 300 second time out, and updated the firmare on them at least one.
The IAPs are all configured the same way that I've got 16 other IAP clusters that are all about the same size (fewer clents though) and the other sites are working fine. I have also updated the firmware on the IAPs at least once during troubleshooting.
Client Match is enabled right now.
11-30-2015 04:48 PM
The APs are pulling their IP addresses from DHCP. I did set the VC IP as a static IP on the cluster, is there any log I could check to see if it is somehow a duplicate?
Thanks for any insight :).
11-30-2015 05:31 PM
Could you try disabling Client Match and see if there is a difference.
Even though Client Match is related to RF, there is more broadcasts (IAP messages) on the wired side seen during Client Match triggers in some cases. There could be some network disruption during that time.
This is not expected when there are no users. However, you can give a try.
11-30-2015 07:12 PM
Yes, DHCP does not give any addresses out in that range. I just changed the IP to another known free address and as soon as the IP change became active nothing started responding on the old address.