Controllerless Networks

Hey all,

 

On hardware controller-based setups, if someone is having a Radius auth issue (or whatever), I can go into the CLI and input the following command(s):

 

logging level debugging user-debug <mac>

 

Then I verify that the supplicant device has been successfully logged:

 

show debug (look for user-debug)

 

And then follow up with show user-table | include <mac>

 

To disconnect the user from the user-table:

 

aaa user delete mac <mac>

 

And then finally these two commands to really dig deep into an issue:

 

show auth-tracebuf mac <mac>

show log user-debug all | include <mac>

 

I can't see to do that on an Instant as the commands are different. Is there a similar way in which to dig into client metrics? I've tried looking at the Support module listed items and can't seem to find anything that looks good.

 

Thank you!

Please go through below link which helps to do user-debug and bunch of other more troubleshooting techinques on Instant.

Hope this helps.

 

http://community.arubanetworks.com/t5/Aruba-Instant-Cloud-Wi-Fi/Troubleshooting-user-connectivity-issues-on-Instant-AP/m-p/79610/highlight/true#M1969

 

http://www.arubanetworks.com/techdocs/InstantMobile/Advanced/Content/Troubleshooting.htm

 

Thank you,

Sriram

Hey Sriram,

 

I wasn't able to get anything useful from any of those commands. Is there anything else that I can try?

 

Thank you!

Acknowledged. If we are trying to troubleshoot 8021.x auth; here is the best way to troubleshoot on IAP

.

Go to support mode from web interface --->choose the option Mulitple AP`s say if you are trying to roam across IAP 1, IAP2 and IAP3 choose all the 3 AP`s. then select below commands.

 

1. show ap association 

2. show ap debug mgmt-frames

3. show ap authentication frames

4. show ap log user-debug

5. show ap log security

 

 then put a filter of your mac-address on the check box and click on auto-run by doing the test on the client couple of times and save the output to the notepad so that we could go back to review the client staus to understand if the client is actually completing 802.11 first and the key derivation and if there is any de-auth from AP and client not responding we could get the reason from output 2 & 3.

 

Could you please attach ap log user-debug to have a look as you had mentioned that didnt help.

Also i would like to know what specific issue we are facing with client to troubleshoot /what specfic we are looking from logs to troubleshoot on IAP?

 

Thank you,

Sriram

 

 

 

Hey Sriram,

 

Thanks for the effort and sorry for the delay. I had to scrub all output before I could post it here. Take a look and let me know what you think. I noticed a lot of MIC errors and found that it could be related to a driver or a certificate, but I wanted to make sure.

 

Thank you!

 

1. AP Association:

MAC MAC MAC y y 1 10 SSID 25 0x0 a-HT-40sgi-2ss 1h:50m:38s 1 W

 

2. AP debug mgmt-frames

May 13 06:32:49 reassoc-resp PORT Success
May 13 06:32:49 reassoc-req PORT -
May 13 06:32:49 auth PORT Success (seq num 3096)
May 13 06:32:49 auth PORT -
May 13 06:32:48 deauth PORT - (internal only)
May 13 06:32:48 deauth PORT Denied; Ageout / MFP-Try Later (seq num 2761)
May 13 06:31:36 reassoc-resp PORT Success
May 13 06:31:36 reassoc-req PORT -
May 13 06:31:36 auth PORT Success (seq num 3054)
May 13 06:31:36 auth PORT -
May 13 06:31:35 deauth PORT - (internal only)
May 13 06:31:35 deauth PORT Denied; Ageout / MFP-Try Later (seq num 2761)
May 13 02:48:20 reassoc-resp PORT Success
May 13 02:48:20 reassoc-req PORT -
May 13 02:48:20 auth PORT Success (seq num 2959)
May 13 02:48:20 auth PORT -
May 13 02:48:20 deauth PORT - (internal only)
May 13 02:46:52 reassoc-resp PORT Success
May 13 02:46:52 reassoc-req PORT -
May 13 02:46:52 auth PORT Success (seq num 2901)
May 13 02:46:52 auth PORT -
May 13 02:46:52 deauth PORT - (internal only)
May 13 02:45:51 reassoc-resp PORT Success
May 13 02:45:51 reassoc-req PORT -
May 13 02:45:51 auth PORT Success (seq num 2879)
May 13 02:45:51 auth PORT -
May 13 02:45:51 deauth PORT - (internal only)
May 12 20:58:08 reassoc-resp PORT Success
May 12 20:58:08 reassoc-req PORT -
May 12 20:58:08 auth PORT Success (seq num 2812)
May 12 20:58:08 auth PORT -
May 12 20:58:08 deauth PORT - (internal only)
May 12 20:56:42 reassoc-resp PORT Success
May 12 20:56:42 reassoc-req PORT -
May 12 20:56:42 auth PORT Success (seq num 2754)
May 12 20:56:42 auth PORT -
May 12 20:56:42 deauth PORT - (internal only)
May 12 20:55:40 reassoc-resp PORT Success
May 12 20:55:40 reassoc-req PORT -
May 12 20:55:40 auth PORT Success (seq num 2687)
May 12 20:55:40 auth PORT -
May 12 20:55:40 deauth PORT - (internal only)
May 12 20:54:36 reassoc-resp PORT Success
May 12 20:54:36 reassoc-req PORT -
May 12 20:54:36 auth PORT Success (seq num 2599)
May 12 20:54:36 auth PORT -
May 12 20:54:36 deauth PORT - (internal only)

 

3. AP Authentication frames

May 13 08:26:49 rad-req -> OMITTED PORT PORT
May 13 08:26:49 rad-accept <- OMITTED PORT PORT
May 13 08:26:49 eap-success <- PORT PORT
May 13 08:26:49 wpa2-key1 <- - PORT
May 13 08:26:49 wpa2-key2 -> - PORT mic failure
May 13 08:26:50 wpa2-key1 <- - PORT
May 13 08:26:50 wpa2-key2 -> - PORT
May 13 08:26:50 wpa2-key3 <- - PORT
May 13 08:26:50 wpa2-key4 -> - PORT
May 13 08:34:21 eap-id-req <- PORT PORT
May 13 08:34:21 eap-id-resp -> PORT PORT OMITTED
May 13 08:34:21 rad-req -> PORT PORT
May 13 08:34:21 rad-resp <- OMITTED PORT PORT
May 13 08:34:21 eap-req <- PORT PORT
May 13 08:34:21 eap-resp -> PORT PORT
May 13 08:34:21 rad-req -> OMITTED PORT PORT
May 13 08:34:21 rad-resp <- OMITTED PORT PORT
May 13 08:34:21 eap-req <- PORT PORT
May 13 08:34:21 eap-resp -> PORT PORT
May 13 08:34:21 rad-req -> OMITTED PORT PORT
May 13 08:34:21 rad-accept <- OMITTED PORT PORT
May 13 08:34:21 eap-success <- PORT PORT
May 13 08:34:21 wpa2-key1 <- - PORT
May 13 08:34:21 wpa2-key2 -> - PORT mic failure
May 13 08:34:22 wpa2-key1 <- - PORT
May 13 08:34:23 wpa2-key2 -> - PORT
May 13 08:34:23 wpa2-key3 <- - PORT
May 13 08:34:23 wpa2-key4 -> - PORT

 

4. AP Log user-debug

May 13 06:27:08 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 06:31:35 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 06:31:35 cli[1580]: <541004> |AP cli| recv_sta_offline: receive station msg, OMITTED.
May 13 06:31:37 cli[1580]: <541004> |AP cli| recv_sta_online: receive station msg, OMITTED.
May 13 06:31:37 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 06:32:49 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 06:32:49 cli[1580]: <541004> |AP cli| recv_sta_offline: receive station msg, OMITTED.
May 13 06:32:50 cli[1580]: <541004> |AP cli| recv_sta_online: receive station msg, OMITTED.
May 13 06:32:51 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 06:40:42 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 06:48:15 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 06:56:18 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 07:03:50 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 07:11:23 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 07:18:55 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 07:26:28 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 07:34:01 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 07:41:33 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 07:49:06 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 07:56:39 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 08:11:44 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 08:19:16 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 08:26:49 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.
May 13 08:34:22 cli[1580]: <541004> |AP cli| recv_stm_sta_update: receive station msg, OMITTED.

 

5. AP log security

May 12 19:51:30 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 12 19:59:03 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 12 20:06:36 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 12 20:14:08 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 12 20:21:41 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 12 20:29:13 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 12 20:36:46 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 12 20:44:19 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 12 20:51:51 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 02:55:56 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 03:03:28 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 03:11:01 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 03:18:33 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 03:26:06 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 03:33:38 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 03:41:11 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 03:48:44 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 03:56:16 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 04:03:49 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 04:11:21 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 04:18:54 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 04:26:27 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 04:33:59 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 04:41:32 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 04:49:04 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 04:56:37 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 05:04:10 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 05:11:42 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 05:19:15 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 05:26:47 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 05:34:20 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 05:41:52 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 05:49:25 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 05:56:58 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 06:04:30 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 06:12:03 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 06:19:35 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 06:27:08 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 06:40:43 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 06:48:20 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 06:56:18 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 07:03:51 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 07:11:23 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 07:26:29 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 07:34:01 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 07:41:34 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 07:49:06 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 07:56:39 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 08:04:11 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 08:11:44 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 08:19:16 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 08:26:49 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED
May 13 08:34:22 stm[1588]: <132094> |AP stm| MIC failed in WPA2 Key Message 2 from Station OMITTED OMITTED OMITTED

Sriram? Anyone wanna take a stab at this? I'm still looking for a reason for this error.

I could notice WPA2 Key 2 is failed which means basically from client to AP.

Key 1 is from AP to client. What type of client we are dealing with & how many clients are affected pls let us know.

Also it would be good to know what happens if you reboot the AP or try connect to different AP to see if we can replicate the issue.

Updating client drivers also could help.
Do the issue occurs both on "a" radio and on "g" radio ? You could try switching radio manually if this client is laptop.

Please make sure RF is clear on terms of channel busy/interference/noise floor on AP/CCI etc....

Testing with different client on same AP also would help to understand more.

Thank you,
Sriram

Hey Sri,

 

Thank you for your reply. I'll address each of your questions one at a time:

 

Key 1 is from AP to client. What type of client we are dealing with & how many clients are affected pls let us know.

We are dealing with one of our sites only, and the device in question is a wireless USB dongle that's connected to desktops. 

Also it would be good to know what happens if you reboot the AP or try connect to different AP to see if we can replicate the issue.

We have 8 APs here, and we have rebooted them (instant cluster), and the problem persists.

Updating client drivers also could help.
Great suggestion and it's one that I preach often, however we have everything managed through our desktop team and they use their own software to ensure that the latest driver is installed, since we also do posture-checking using ClearPass to probihit connectivit to the main VLAN onsite if the device doesn't meet our requirements (the newest driver being installed as one).

Do the issue occurs both on "a" radio and on "g" radio ? You could try switching radio manually if this client is laptop.

We use band steering to prefer a 5ghz connection to the AP, but it's been observed at both 2.4ghz and 5ghz.

Please make sure RF is clear on terms of channel busy/interference/noise floor on AP/CCI etc....

All APs are transmitting at the appropriate power level (per Airwave and CLI), and we have identified no other sources of non-wifi interference at this location eating up channel space. Noise floor and CCI are all within norms.

Testing with different client on same AP also would help to understand more.

We have a mixed client environment there, and this is only affecting the above listed device.

 

I feel as though there's something that was changed inadvertently and I just need to locate it. Perhaps it has something to do with checking to make sure our other teams have really updated the driver and modified the power settings on the desktops to prevent the USBs from shutting down, or letting Windows regulate power output from USB attached devices like they're supposed to have done. Anyway, I've run out of ideas and even after this full tracebuf and auth output, I'm still stuck.

 

Thank you!