Controllerless Networks

Hello everyone,

 

We experience problems with random clients (laptops, smartphones) suddenly loosing wifi connectivity. Some of the clients are on the supported device list, others are not.

 

During a connection period of about 4 hours, a random client might have 4 or 5 connection problems at which they loose up to 10 pings. Other clients might or might not experience that problem when connected to the same network, on the same location.

 

Our setup is made of 9 Aruba IAP-115 Instant AP's with one acting as a virtual controller.

they are running InstantOS 4.1.1.1 and are managed by the Airwave platform 8.0.5.

 

In order to get this problem solved, we contacted our supplier. During their visit they told us that Instant APs have no real debugging features to log or trace these kind of intermittent problems. I find that hard to believe. (they suggest we bring in a controller, convert our APs to campus APs and check if the problem is still there. Then activate logging on the controller.)

 

Is there any command I can run on the Instant controller to debug these intermittent problems?

The problem is that we can not simulate the problem whenever we want. 

 

Any help on this is appreciated.

 

Kind regards,

 

Tim of BICT NetworkAdmins

 

BICT,

 

I would make sure "Broadcast Filtering" is set to "ALL"  http://www.arubanetworks.com/techdocs/Instant_41_WebHelp/InstantWebHelp.htm#UG_files/WLAN_SSID_conf/ConfiguringWLANPro.htm%3FTocPath%3DWireless%20Network%20Profiles%7CConfiguring%20Wireless%20Network%20Profiles%7C_____1

 

Also, I would start with you minimum transmit power be 12 and your maximum transmit power be 18 under ARM settings:  http://www.arubanetworks.com/techdocs/Instant_41_WebHelp/InstantWebHelp.htm#UG_files/ARM/Configuring_ARM-Features.htm%3FTocPath%3DAdaptive%20Radio%20Management%7C_____2

 

I would start by trying out both of those, first...

 

Broadcast filter all would improve performance by suppressing uneeded broadcasts

 

Having a transmit power min of 12 and max of 18, would match the power of your clients and possibly allow them to roam better...

Hi Colin,

Thank you for your suggestions.
I will implement these settings and get back to you.

Kind regards,
Tim De Coen Beaulieu ICT

Hi Colin,

 

After implementing these settings, we still had a lot of problems with the mobile clients dropping connections. I disabled all 802.11 options for the SSID (802.11r, 802.11k, 802.11v) and it looks like things are better.

 

The only problem is that now is a difficult time to test because lots of people are out of the office during the holidays.

 

Could the "inactivity timeout" option under the SSID have something to do with the problems? By default it's set to 1000 sec.

What will happen to the client once the timer expires?

 

Kind regards,

 

Tim

802.11r, 802.11k and 802.11v are not enabled by default and can cause issues for clients that do not support it.  You might have solved your own problem.

UPDATE 6/2018 -  The updated RF and Roaming Optimization Validated Reference Design Guide (VRD) has been published and has updated recommendations about enabling 802.11v, k and r in user networks.  The VRD can be found here: http://community.arubanetworks.com/t5/Validated-Reference-Design/RF-and-Roaming-Optimization-for-Aruba-802-11ac-Networks/ta-p/432994

 

I would not touch the inactivity timer, because (1) it would deal mainly with inactivity, which is not your problem and (2) you should not make more than one major change at a time.

Hello,

 

We have been testing alot with multiple people around the company during these last days and

it seems that the problems have decreased significantly since the last change.

 

However, there are still some clients that have problems at random.

Is there some way (logging/ debugging command) we can see on the IAP setup what's happening or going wrong?

 

Kind regards,

 

Tim De Coen

 

BICT,

 

Now is the time to get specific information about these clients:

 

- What OS and adapter is the network card on the client?

- What is the driver date?

- Is that SSID using encryption?

 

Does IDS detection is ON? (If so , u may do under CLI - " show log security " and check if there is any attacks on your Wi-Fi enviroment / clients.

More log commands:

http://www.arubanetworks.com/techdocs/Instant_40_Mobile/Advanced/Content/CLI_commands/Accessing%20Instant%20CLI.htm

 

Does ClientMatch feature enabled?

If so - try to disable this option - and share with us the results.

 

kdisc98,

 

What is the technical reason for disabling clientmatch and how do you measure the results when you do that?  The poster said he still had problems with only a few clients, so disabling clientmatch will remove that advantage from ALL clients, not just the few that are having problems.  If the user is having specific problems with a few clients, the issue could be with those clients.

cjoseph.

 

It's an idea - to invstigate another cause. **TRY DISABLE The clientmatch and let us know if the problem is gone**

 

It's just to understand if the cause is from Client match (is was encouter a few times that client match cause clients to feel real disconnect from the Wi-Fi )

 

I agree with you,(it's sound like a client side issue) Some mobile devices/ Computer chipsets (old drivers)  dont like clientmatch enviroment.

 

 

---

@kdisc98 wrote:

cjoseph.

 

It's an idea - to invstigate another cause. **TRY DISABLE The clientmatch and let us know if the problem is gone**

 

It's just to understand if the cause is from Client match (is was encouter a few times that client match cause clients to feel real disconnect from the Wi-Fi )

 

I agree with you,(it's sound like a client side issue) Some mobile devices/ Computer chipsets (old drivers)  dont like clientmatch enviroment.

 

 

---

kdisc98,

 

Thank you.

Hello kdisc98,

 

Thank you for joining this discussion in order to help us finding a solution.

 

We updated all drivers of our clients yesterday morning. 

We are investigating the situation now and if we still have problems reported by the end if today, then I will try to disable client match tomorrow morning and evaluate.

 

IDS is not enabled.

 

 

Hello cjoseph,

 

We have clients running windows 7 and windows 8.1 using intel centrino N-6205 wifi adapters.

All drivers are up to date. version 15.13 for W7 and 15.14 for W8.1

 

The problems we are seeing now is that clients suddenly get disconnected. When trying to reconnect, it just keeps trying. When disabling/enabling the wifi adapter or connecting to another SSID and then back again to the correct SSID, the client is connected again.

 

We also see this on our corporate phones (Samsung S3 mini, Samsung S4 mini)

 

To authenticate corporate laptops, we use 802.1x with client certificates connecting to external radius servers using the dynamic radius proxy configuration on the controller. 

 

To authenticate corporate phones, we use AD account/password with PEAP MS-CHAPv2

 

Hope this is the info you are looking for.

 

Kind regards,

 

Tim De Coen

 

 

BICT,

 

How many SSIDs do you have enabled?

What is your backend radius server?

Are there any specific areas where the problem happens?

What is the maximum simultaneous users that you see on your network during the day?

Is the problem localized to a specific area?

In the RF dashboard, what is the RF Utilization of that specific access point?

 

 

Hello cjoseph,

 

We have 3 SSID's enabled. One for production, offices and visitors.

Our backend radius servers are Windows 2008 R2 domain controllers running the latest windows patches.

 

It seems that the problems are not location related.

We have at this moment 86 clients connected to 8 AP's with a max of 28 clients to a single AP (moving 1.39 Mbps of traffic)

 

While checking the RF Dashboard, I noticed an Alert.

 

 

Does this mean that clients can sometimes not reach the radius server?

Or could this be authentication failures from clients?

 

Kind regards,

 

Tim De Coen

The Alert (Number 80 in number. of alerts you had) Seems like there is connecctivity issues between your radius and your DRP/VirtualIP address.

This might causing your client's not be able to re-auth

you may run the command: (via cli)

show log secuirty

or

show log system

in order to see more / detail alerts.

This is what the system log shows. I have some concerns about the following messages:

 

Jan 6 12:37:58 cli[1751]: <341004> <WARN> |AP OROAP101@10.131.99.11 cli| AP 10.131.99.15: Client e0:b5:2d:ec:2c:fd authenticate fail because RADIUS server connection failure

Jan 6 12:38:58 cli[1751]: <341004> <WARN> |AP OROAP101@10.131.99.11 cli| AP 10.131.99.15: Client a8:86:dd:67:f0:42 incorrect vlan config

 

Jan 6 12:49:01 cli[1751]: <341004> <WARN> |AP OROAP101@10.131.99.11 cli| AP 10.131.99.15: Client e8:b1:fc:18:0e:7a authenticate fail because integrity check failure in encrypted message

 

I see these messages comming back alot. But I wonder what would be the problem as not everyone is complaining. These have to be intermittent problems and not configuration problems then?

 

Tim

Authenticate fail because RADIUS server connection failure =  IAP(VC) VirtualIP/DRP isnt able to comminucte with your Radius (secret/network/vlan)

 

Are u aware that Native VLAN = Default VLAN (no tagging is needed)

 

 

Integrity check failure in encrypted message


The IAP cannot receive data from this client , because the integrity check of the received message (MIC) has failed.


Check the encryption setting on the client and on the IAP.

What do you mean with that no tagging is needed.

Our access point uses 4 vlans. 

- management of AP's = vlan 99

- production SSID = production VLAN

- office SSID = office VLAN

- visitors SSID = visitors VLAN

 

All VLANs are tagged on the switch port.

In the AP config, I tell the AP to send data on vlan 99

DRP vlan for the controller is also configured for vlan 99

Our access point uses 4 vlans. 

- management of AP's = vlan 99

 

1.Your InstantAP'S are communitcting via this VLAN (99) and sending the radius request via this native vlan(v99)

 

 

This is importent = VLAN99 should be untagged (native vlan) on all your instantap switch ports.

 

If you choose to transmit this VLAN on-air(in ssid) dont tag it as v99 , choose it as Default VLAN

 

2.Another thing, In your Radius,u inserted the DRP IP (VirtualIP) as NAS-IP right?

---

@BICT Networkadmins wrote:

Hello cjoseph,

 

We have 3 SSID's enabled. One for production, offices and visitors.

Our backend radius servers are Windows 2008 R2 domain controllers running the latest windows patches.

 

It seems that the problems are not location related.

We have at this moment 86 clients connected to 8 AP's with a max of 28 clients to a single AP (moving 1.39 Mbps of traffic)

 

While checking the RF Dashboard, I noticed an Alert.

 

 

Does this mean that clients can sometimes not reach the radius server?

Or could this be authentication failures from clients?

 

Kind regards,

 

Tim De Coen

---

Tim,

 

Make sure you have Broadcast Filltering set to ALL for  all 3  of your SSIDs.  This is important

 

Is the ORODC003 the only radius server that shows down intermittently?  If it is the only one that seems to have problem, consider removing  it as a radius server and only use one.  

If both radius servers show that they are down, we need to look at the path between the Instant APs and the radius server:  

How are the instant APs powered?  

Can you look at the switchports that the instants are connected to and see any errors, crcs, etc?  

Could they have possibly negotiated to the wrong duplex and speed?  

Can you take a look at the radius server(s) to see what is their CPU utilization in task manager?  

Are you using the radius server(s) for anything else?  

Can you take a look at your radius server ports on the switch to make sure there are no duplex errors or mismatches?

 

The radius servers showing out of service is a serious issue that needs to be resolved.  That should not be happening on a reliable wired network.

 

 

@cjoseph,

 

If all SSID's are linked to different VLANs, then they are not in the same broadcast domain.

Is the Broadcast filtering still important then? for now it's only enabled for the office SSID as this is where the problems occur.

 

We have 2 radius servers configured, but only the primary is showing errors.

The strange thing is, that the primary one is located onsite and the backup is in our datacenter.

 

I will disable the primary radius server.

 

Some of the AP's are powered by PoE modules (not via the switches) and others are connected on a PoE enabled switch.

We do not see errors on the switchports, we checked this.

 

Concerning the radius servers. this are the domain controllers so I guess they will be used by other processes as well and not only for radius.

 

server values are: CPU = 0%, memory= 45%, NIC load =0.05% (1 Gbps)

and the radius server is a virtual machine. so no duplex mismatches are possible.

 

 

---

@BICT Networkadmins wrote:

@cjoseph,

 

If all SSID's are linked to different VLANs, then they are not in the same broadcast domain.

Is the Broadcast filtering still important then? for now it's only enabled for the office SSID as this is where the problems occur.

 

We have 2 radius servers configured, but only the primary is showing errors.

The strange thing is, that the primary one is located onsite and the backup is in our datacenter.

 

I will disable the primary radius server.

 

Some of the AP's are powered by PoE modules (not via the switches) and others are connected on a PoE enabled switch.

We do not see errors on the switchports, we checked this.

 

Concerning the radius servers. this are the domain controllers so I guess they will be used by other processes as well and not only for radius.

 

server values are: CPU = 0%, memory= 45%, NIC load =0.05% (1 Gbps)

and the radius server is a virtual machine. so no duplex mismatches are possible.

 

 

---

BICT,

 

In wireless, the RF is the broadcast domain because it is shared.  If a device on VLAN4 sends traffic,  no other devices on that access point, regardless of VLAN can send traffic at the exact same time.  If you enable broadcast filtering on all SSIDs it will remove that phenomenon for useless broadcasts and improve performance.

 

I would look in the event viewer on the server to see if I see any errors, as well.

 

I disabled the primary radius server and I see the same problems popping up for the secundary radius server.

 

So the problem will not be related to a single radius server.

 

@ cjoseph,

 

I will enable broadcast filtering on all SSID's

 

@kdisc98,

 

I will change the configuration of the management vlan and keep you posted.

Hello,

 

The vlan configuration is now changed for all IAPs, all are rebooted and the alerts are cleared.

For now, we see no new alerts. We will start testing tomorrow morning when people arrive back in the office.

 

I will keep you posted.

 

Kind regards,

 

Tim De Coen

No alerts its Good,keep us updated, what solve your issue ( the disconnect connect issue )

We still get these messages on a regular basis. each 10-15 min

 

"Jan 7 08:56:44 cli[1739]: <341004> <WARN> |AP OROAP101@10.131.99.11 cli| AP 1 0.131.99.14: Client 34:e2:fd:b7:ed:07 authenticate fail because RADIUS server co nnection failure

 

 

You talked about the DRP configuration. If the native vlan is set to 99, what should the DRP-vlan setting be?

 

In your case (if im not wrong)

Native VLAN - Untagged (in your switch) = V99 = Default Vlan

SO DynaicRadiusProxy VLAN should be 0 = default VLAN = VLAN 99 (native in your case)

The DRP-vlan is configured with 0.

 

I see that during the morning 74 clients connected to the network.

During that timeframe (1.5 hours) we got 7 messages that clients where not able to connect due to connection failure.

 

I have a ping running from the radius server to the controller which shows no timeouts.

 

The connection problems (disconnects) are still being evaluated.

 

Any ideas about the radius connection failures?

Hi Everyone,

 

The stability of the wifi network has improved significantly over the last couple of weeks.

We notice however that still some clients are experiencing these intermittent connection problems.

We were able to match these events to the Radius errors.

 

We have no network related issue between controller and Radius server.

We installed a new W2K8 server which is only used for Radius.

 

We get an average of 35 messages a day.

 

Does anyone have any idea how we might be able to solve this?

 

Thanks a lot

You should open a support case to get to the bottom of this.  You have new information about radius errors, and that could help them get to the root cause.