I did not, but I will connect my switch to my syslog server to be sure.
But it seems unlikely to have the same problems on two different access points (connected through two separate physical links). And I monitor my IOT devices on my network, and if it was a physical cable problem to an access point I would see all the devices going down at the same time I'm having problems on my phone I would imagine.
And the AP's are powered though the physical link with POE, so if it went down the complete AP would shutdown also I think, and then I would defnitely notice.
Original Message:
Sent: Feb 15, 2025 02:29 PM
From: shpat
Subject: Constant disconnections from AP635 virtual controller network
Did you try doing debug on the switch and check if there is Layer-1 issue potentially (physical cable) ? I know it sounds a bit odd but maybe you should consider that as well.
------------------------------
Shpat | ACEP | ACMP | ACCP | ACDP
Just an Aruba enthusiast and contributor by cases
If you find my comment helpful, KUDOS are appreciated.
Original Message:
Sent: Feb 15, 2025 12:34 PM
From: Sjoerdfc
Subject: Constant disconnections from AP635 virtual controller network
Hi shpat,
If I check the power consumption, it's only 8.1 and 8.6 watts on the ports of the switch. And in total my switch is using 24W, so it has some headroom (it can deliver 60 watts total).
I checked the datapath session table, these are some of the entries I saw. Nothing with my phones MAC, only it's IP:
Source IP Destination IP Prot SPort Dport Cntr Prio ToS Age Destination TAge Packets Bytes Flags Offload flags
---------------- -------------- ---- ----- ----- ---- ---- --- --- ----------- ---- ------- ----- ------ -------------
10.0.0.110 255.255.255.255 17 46375 21027 0 0 0 0 dev27 3 1 f3 FC
10.0.0.110 10.0.0.4 17 42749 53 0 0 0 0 dev27 19 5 1b8 FCI
255.255.255.255 10.0.0.110 17 21027 46375 0 0 0 0 dev27 3 0 0 FY
I already disabled all the channels except one non-DFS 20MHz channel a while ago, and now I'm running with all the channels except DFS checked. Only 20/40 or 80 doesn't seem to make a difference.
I find it very unlikely that this is an Android sleep problem, because I'm actively using my phone when it happens (browsing, viewing security cam footage, SSH'ing into a machine) and suddenly it disconnects. It's very noticable with SSH for example because it instantly disconnects. But sometimes it's also so bad I can't even browse websites because it keeps disconnecting every minute, and the internet drops, and it needs to connect again.
The 'show ap arm status' is good (status, state doesn't exists on my AP's), I will try and run that when I'm having problems and compare results, to see which variable changes.
Original Message:
Sent: Feb 15, 2025 02:36 AM
From: shpat
Subject: Constant disconnections from AP635 virtual controller network
You can briefly check a few things on here:
- The SG350 supports 802.3at (should be 30W capable ports) and the AP-635 in worst case scenario can consume up to 23.8W. Check "show power inline" if there is sufficient power dragged by AP.
- On the AP CLI, check the datapath session table | including the users mac address and check if there is anything you can see in the logs
- Disable the DFS Channels because there might be issues with Radar-detect (happened to a lot of end customers on my end). Also, try to disable 80MHz and enable only 40MHz and test it.
- Check the ARM State : show ap arm state
- Check if there is any issue with Android entering what they call "sleep mode" which usually ends up with disconnecting from the WiFi. You should disable 802.11k/v/r
- You can also try to debug client trace using clients-mac for testing/log gathering
-
------------------------------
Shpat | ACEP | ACMP | ACCP | ACDP
Just an Aruba enthusiast and contributor by cases
If you find my comment helpful, KUDOS are appreciated.
Original Message:
Sent: Feb 15, 2025 01:13 AM
From: Sjoerdfc
Subject: Constant disconnections from AP635 virtual controller network
Both AP-635's are connected to the same Cisco SG350 switch which provides 803.11at POE. I forgot to mention I had IPM disabled in the beginning, but it's enabled now. So that doesn't seem to make a difference.
All my wireless networks are different VLAN's, and these VLAN's are tagged on the switchport. So in the Wifi network configuration I have the 'Client VLAN assignment' set to static and their respective VLAN. I have the uplink port of the AP's configured with the management VLAN, which is the same VLAN as the personal Wifi network.
DHCP is provided by my OPNsense router.
At first I was on firmware 8.12.0.4, and then I downgraded to 8.10.0.13. Now I'm on 8.12.0.2.
Original Message:
Sent: Feb 14, 2025 11:46 PM
From: ariyap
Subject: Constant disconnections from AP635 virtual controller network
so you have 2x AP-635s (one upstairs and one downstairs) that are connected to the same LAN switch?
Is the LAN switch providing PoE to the APs?
are your 3x WLANs (Guest/IoT/Personal) are mapped to different VLANs or the same? and is it the same VLAN as that of the APs?
Is the LAN switch providing DHCP services as well?
what were the exact firmware versions in 8.12.x and 8.10.x that you used?
------------------------------
If my post was useful accept solution and/or give kudos.
Any opinions expressed here are solely my own and not necessarily that of HPE or Aruba.
Original Message:
Sent: Feb 13, 2025 08:34 AM
From: Sjoerdfc
Subject: Constant disconnections from AP635 virtual controller network
Hi,
I'm hoping someone can help me. I'm almost at the point of throwing away my Aruba access points.
I have a Virtual Controller cluster of two AP-635 (downstairs and upstairs) access points. I have a Guest network (5Ghz), a IOT network (2.4 only) and a personal network (5Ghz) all on different vlans. My phone (Android) constantly keeps disconnecting from the wifi (and other people on my wifi complain as well). I understand that if I move from downstairs to upstairs it reconnects to the other AP. But if I don't move it also constantly disconnects for very short periods of time. Sometimes only half a second, sometimes seconds. Sometimes this doesn't happen for half an hour, sometimes it happens so often I can't browse websites because it disconnects right before I press enter to open a website.
The floors of the house are concrete, and I disabled one of the access points for testing. When I'm upstairs the downstairs AP is practically not in range and vice versa. So it seems I'm not actually roaming accidentally between AP's. I also checked the wifi channel I'm on, and it doesn't change after a disconnection, my phone says on the same channel.
I have tried:
- AOS 8.12 and 8.10 (specifically 8.10.0.13 because of another discussion about hiccups with newer versions, but that seems like a different issue entirely)
- Turning Client Match on and off
- Turning v/k/r on and off in different combinations
- Turning band steering on and off
- Limiting the transmit power to different levels
- Turning off High throughput and High efficiency
- All the Broadcoast filtering settings
- Turning wide channels/80Mhz off
- Switched off all channels except 36 (so no DFS)
- Switched from WPA3 to WPA2
- Factory reset the AP's and configured everything from scratch
- I bought a new phone (also Android) unrelated to these problems and it has same problem
I had the same router (OPNSense) with practically the same configuration in my old house with Ubiquity access points and it worked without problems.
I checked with "show ap debug auth-trace-buf mac", and sometimes I see my phone re-authenticating, but most times I see nothing at all.
I enabled syslog and some debug logging, and here is a relevant part of the syslog:
2025-02-11 13:37:09.000 10.0.0.952025 10.0.0.95 sapd[8546]: <404014> <DBUG> AP:aruba-ap-beneden <10.0.0.95 DC:B7:AC:C1:03:2C> AM: FTM: randomize_scan, 5056 ftm flag is disable2025-02-11 13:37:11.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: stm_send_sta_offline: Sending sta offline msg to CLI0, mac='b8:98:ad:0f:76:ad'2025-02-11 13:37:11.000 10.0.0.922025 10.0.0.92 cli[8528]: <541004> <WARN> AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> recv_sta_offline: receive station msg, mac-b8:98:ad:0f:76:ad bssid-dc:b7:ac:8c:72:f0 essid-PersonalWifi timestamp-1739277431-300707.2025-02-11 13:37:11.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: stm_update_sap_num_client:26296, bssid dc:b7:ac:8c:72:f0, num_client 02025-02-11 13:37:11.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: AP 127.0.0.1, vap dc:b7:ac:8c:72:f0, num_client 02025-02-11 13:37:11.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: stm_send_sta_offline: Sending sta offline msg to CLI0, mac='b8:98:ad:0f:76:ad'2025-02-11 13:37:11.000 10.0.0.922025 10.0.0.92 cli[8528]: <541004> <WARN> AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> recv_sta_offline: receive station msg, mac-b8:98:ad:0f:76:ad bssid-dc:b7:ac:8c:72:f0 essid-PersonalWifi timestamp-1739277431-302243.2025-02-11 13:37:11.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> sapd[8547]: sapd_proc_stm_reset_key: Got STM Reset key bss=dc:b7:ac:8c:72:f0 mac=b8:98:ad:0f:76:ad, idx=02025-02-11 13:37:12.000 10.0.0.952025 10.0.0.95 AP:aruba-ap-beneden <10.0.0.95 DC:B7:AC:C1:03:2C> sapd[8546]: sapd_esl_usb_link_monitor_timeout: ESL-Radio not plugged.2025-02-11 13:37:13.000 10.0.0.952025 10.0.0.95 sapd[8546]: <404014> <DBUG> AP:aruba-ap-beneden <10.0.0.95 DC:B7:AC:C1:03:2C> AM: FTM: randomize_scan, 5056 ftm flag is disable2025-02-11 13:37:13.000 10.0.0.922025 10.0.0.92 sapd[8547]: <404014> <DBUG> AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> AM: FTM: randomize_scan, 5056 ftm flag is disable2025-02-11 13:37:13.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: send_register_local,member send defaultcert checksum at heartbeat,cs_defaultcert_csum= 42125751852025-02-11 13:37:13.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: recv_heartbeat_local,compare defaultcert checksum,cs_defaultcert_csum= 4212575185 ,received defaultcert_csum =42125751852025-02-11 13:37:13.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: recv_heartbeat_local, AP(127.0.0.1) config has taken effect2025-02-11 13:37:13.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: receive ap 127.0.0.1 with drt status 02025-02-11 13:37:13.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: swarm_timer_handler,check_ssh2025-02-11 13:37:14.000 10.0.0.942025 10.0.0.94 AP:aruba-ap-zolder <10.0.0.94 34:3A:20:CF:11:08> cli[4860]: arping 2939 times to 10.0.0.1 and arp_result is 0x02025-02-11 13:37:14.000 10.0.0.942025 10.0.0.94 AP:aruba-ap-zolder <10.0.0.94 34:3A:20:CF:11:08> cli[4860]: send_register_local,member send defaultcert checksum at heartbeat,cs_defaultcert_csum= 42125751852025-02-11 13:37:14.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: recv_heartbeat_local,compare defaultcert checksum,cs_defaultcert_csum= 4212575185 ,received defaultcert_csum =42125751852025-02-11 13:37:14.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: recv_heartbeat_local, AP(10.0.0.94) config has taken effect2025-02-11 13:37:14.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: receive ap 10.0.0.94 with drt status 02025-02-11 13:37:14.000 10.0.0.942025 10.0.0.94 AP:aruba-ap-zolder <10.0.0.94 34:3A:20:CF:11:08> cli[4860]: swarm_timer_handler,check_ssh2025-02-11 13:37:17.000 10.0.0.922025 10.0.0.92 sapd[8547]: <404014> <DBUG> AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> AM: FTM: randomize_scan, 5056 ftm flag is disable2025-02-11 13:37:18.000 10.0.0.942025 10.0.0.94 sapd[4877]: <404014> <DBUG> AP:aruba-ap-zolder <10.0.0.94 34:3A:20:CF:11:08> AM: FTM: randomize_scan, 5056 ftm flag is disable2025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: AP 127.0.0.1, vap dc:b7:ac:8c:72:f0, num_client 12025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: stm_update_sap_num_client:26296, bssid dc:b7:ac:8c:72:f0, num_client 12025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 cli[8528]: <541004> <WARN> AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> recv_sta_online: receive station msg, mac-b8:98:ad:0f:76:ad bssid-dc:b7:ac:8c:72:f0 essid-PersonalWifi timestamp-1739277438-820518.2025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: Backup roaming info: bssid: dc:b7:ac:8c:72:f0, snr: 51, rssi: 41, channel: 116E, auth time: 17392762522025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: derive_post_auth_acl_by_named_role: 156: derive role by name PersonalWifi, acl 153, and derive vlan 0 from vlan2025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: VLAN_HIGHER_PRECEDENCE_THAN_STORED: 1339: vlan_rule_index=ff, sap_sta->vlanhow=ff, precedence_result=12025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: __HIGHER_PRECEDENCE_COMPARE: 1316: matched_rule_index=a7fff, sap_sta->acl_rule_index=0, precedence_result=12025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 cli[8528]: <541004> <WARN> AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> recv_stm_sta_update: receive station msg, mac-b8:98:ad:0f:76:ad bssid-dc:b7:ac:8c:72:f0 essid-PersonalWifi timestamp-1739277438-822634.2025-02-11 13:37:18.000 10.0.0.942025 10.0.0.94 AP:aruba-ap-zolder <10.0.0.94 34:3A:20:CF:11:08> cli[4860]: Receive session mobility req for client b8:98:ad:0f:76:ad2025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: recv_stm_sta_update set client b8:98:ad:0f:76:ad connect status: 1, through ip: 10.0.0.110, acl: 1532025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: stm_send_sta_update: Sending sta update msg to CLI0, mac='b8:98:ad:0f:76:ad'2025-02-11 13:37:18.000 10.0.0.952025 10.0.0.95 AP:aruba-ap-beneden <10.0.0.95 DC:B7:AC:C1:03:2C> cli[8527]: Receive session mobility req for client b8:98:ad:0f:76:ad2025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: rap_bridge_user_handler: 16236: user entry deleted for '10.0.0.110' 'b8:98:ad:0f:76:ad'2025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: PAPI_Send: sendto Authentication failed: (null) Message Code 24588 Sequence Num is 424392025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 stm[8557]: <304065> <ERRS> AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> PAPI_Send failed, send_papi_message_with_args, 1355: No such file or directory, dstport 56642025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> sapd[8547]: sapd_proc_stm_keys:klen=16, bssid=dc:b7:ac:8c:72:f0, mac=b8:98:ad:0f:76:ad, standby=02025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> sapd[8547]: sap_recv_keys: default type=308162025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> sapd[8547]: do_madwifi_set_key:isgroup=0 isigtk=0 dev=aruba000 bssid=dc:b7:ac:8c:72:f0 klen=16 idx=655352025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> sapd[8547]: sapd_proc_stm_txkey: Got STM Tx key msg2025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: asap_update_client_name:16991 client b8:98:ad:0f:76:ad name updated to Edge202025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: recv_sta_update_dhcp_ops, recv sta:b8:98:ad:0f:76:ad, dhcp_option12:Edge20, dhcp_option55:0103060F1A1C333A3B2B726C, dhcp_option60:android-dhcp-15, dhcp_options:61,50,57,60,12,552025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: rap_bridge_user_handler: 16327: rbs update: flags:1/2 aclnum:153 ip:10.0.0.110 mac:b8:98:ad:0f:76:ad bssid:dc:b7:ac:8c:72:f0 vlan:20 wired:02025-02-11 13:37:18.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> cli[8528]: recv_user_sync_message set client b8:98:ad:0f:76:ad connect status: 1, through ip: 10.0.0.110, acl: 1532025-02-11 13:37:19.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: Recv dns state update, state: 12025-02-11 13:37:19.000 10.0.0.922025 10.0.0.92 AP:aruba-ap-boven <10.0.0.92 DC:B7:AC:C0:C7:2F> stm[8557]: Recv dns state update, state: 2
The MAC address of my phone is b8:98:ad:0f:76:ad, and it's IP is 10.0.0.110. In the log it was connected to "aruba-ap-boven" (10.0.0.92). Every time it happens I see the same messages, 'asap_update_client_name' and then 'recv_sta_update_dhcp_ops'. And also sometimes 'user entry deleted'.