Wireless Access

Reply
MVP

Random Drops For Whole Site

Hi,

 

We have been experiencing random drops in connectivity for a whole site a couple of times a day.  I noticed in the logs that it is showing the tunnel is going down and up but not sure how further to troubleshoot what might be the cause - can any one help? Logs below:


Virtual Mobility Master running Aruba OS 8.2.2.5

 

(BIRMINGHAM-AT-2) #show log system all | include TUN

Aug  7 14:47:03  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug  8 10:14:55  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug  8 10:15:01  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug  9 08:50:52  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug  9 08:50:59  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug  9 09:51:06  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug  9 09:51:11  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug 11 04:55:55  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug 11 04:56:18  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug 12 03:09:04  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug 12 03:09:21  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug 13 08:51:36  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug 13 08:51:53  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug 13 14:19:38  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug 13 14:20:01  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug 14 08:04:09  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug 14 08:04:33  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug 14 11:01:01  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug 14 11:01:24  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Aug 14 11:06:37  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap

Aug 14 11:07:06  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0

Guru Elite

Re: Random Drops For Whole Site

On the master, type "show master-local stats" to see if any heartbeats are indeed being dropped between the master and the MDs.


*Answers and views expressed by me on this forum are my own and not necessarily the position of Aruba Networks or Hewlett Packard Enterprise.*
ArubaOS 8.4 User Guide
InstantOS 8.3 User Guide
Airheads Knowledgebase
Airheads Learning Videos
Aruba Central Documentation
Sign up for Security Alerts
Aruba Technical Webinars
MVP

Re: Random Drops For Whole Site

On the active controller on that site? 

 

(BIRMINGHAM-AT-2) #show master-local stats

 

Wed Aug 14 14:07:05.116 2019

 

Missed -> HB Resp from Master

-----------------------------

IP Address    HB Req      HB Resp     Cfg Terminate  Peer Reset  Total Missed  Last Sent Missed  Last Synced/First Missed

----------    ------      -------     -------------  ----------  ------------  ----------------  ------------------------

192.168.12.4  60135       60144       0              58          268           0                 Wed Aug 14 13:02:48 2019

(BIRMINGHAM-AT-2) #

 

On the Mobility Master

 

(Mobility-Master-Primary) [mynode] #show master-local stats

 

Missed -> HB Req from Local(s)

------------------------------

IP Address    HB Req      HB Resp     Cfg Terminate  Peer Reset  Total Missed  Last Sent Missed  Last Synced/First Missed

----------    ------      -------     -------------  ----------  ------------  ----------------  ------------------------

172.20.0.202  128847      128847      0              0           0             0                 Pending

192.168.22.3  121227      121410      0              32          297           0                 Wed Aug 14 14:03:13 2019/Wed Jul 31 15:01:28 2019

192.168.37.4  113213      113398      0              2           2             0                 Wed Aug 14 14:03:22 2019/Fri Aug  2 11:05:04 2019

192.168.37.3  103500      103552      0              2           2             0                 Wed Aug 14 13:03:10 2019/Fri Aug  9 10:07:59 2019

192.168.12.4  60060       60188       0              15          16            0                 Wed Aug 14 13:02:48 2019/Wed Aug 14 12:08:01 2019

192.168.12.3  44204       44230       0              11          11            0                 Wed Aug 14 12:57:57 2019/Wed Aug 14 12:07:02 2019

(Mobility-Master-Primary) [mynode] #

 

 

 

Guru Elite

Re: Random Drops For Whole Site

Last drop was around noon today.  I would do a "clear master-local stats" to reset the counters and see how they grow in the next 24 hours.

 

MDs (controllers) do not need persistent connectivity with the MM.


*Answers and views expressed by me on this forum are my own and not necessarily the position of Aruba Networks or Hewlett Packard Enterprise.*
ArubaOS 8.4 User Guide
InstantOS 8.3 User Guide
Airheads Knowledgebase
Airheads Learning Videos
Aruba Central Documentation
Sign up for Security Alerts
Aruba Technical Webinars
MVP

Re: Random Drops For Whole Site

Clear master local stats on the controller itself or on the mobility master?

 

Also do you think maybe this is just showing the connection goes down between the controller and the MM - but that something else is causing the drop to the whole site as well as causing the issue with the communication to MM?

 

Is there anything else i should be checking for in the logs ? I noticed this:

Aug 14 11:01:25 :399816:  <3433> <ERRS> |cfgm|  handle_read: State(READY:UPDATE SUCCESSFUL:CFGID-264:PEND-246:INITCFGID:0) FD=27:Failure receiving heartbeat response header information Result=-1 Err=Connection reset by peer

Aug 14 11:01:25 :399816:  <3433> <ERRS> |cfgm|  Starting BOC log collection timer

Aug 14 11:01:32  KERNEL(Birmingham-AP03-23367@192.168.12.18): [486986.580770] VAP device aruba000 created osifp: (dbe4a540) os_if: (d9e68000)

Aug 14 11:01:32  KERNEL(Birmingham-AP03-23367@192.168.12.18): [486986.635847] VAP device aruba001 created osifp: (dbe4b540) os_if: (dd2b8000)

Aug 14 11:01:32  KERNEL(Birmingham-AP04-23368@192.168.12.14): [486987.080395] VAP device aruba000 created osifp: (dc73b540) os_if: (da450000)

Aug 14 11:01:32  KERNEL(Birmingham-AP04-23368@192.168.12.14): [486987.135472] VAP device aruba001 created osifp: (dc73f540) os_if: (dcdc8000)

Aug 14 11:01:32  KERNEL(Birmingham-AP04-23368@192.168.12.14): [486987.229536] VAP device aruba100 created osifp: (dce84540) os_if: (dbe68000)

Aug 14 11:01:33  KERNEL(Birmingham-AP04-23368@192.168.12.14): [486987.284738] VAP device aruba101 created osifp: (dce81540) os_if: (dbe28000)

Aug 14 11:01:33  KERNEL(Birmingham-AP06-23364@192.168.12.17): [486986.284457] VAP device aruba000 created osifp: (d9567540) os_if: (d8520000)

Aug 14 11:01:33  KERNEL(Birmingham-AP03-23367@192.168.12.18): [486987.629911] VAP device aruba100 created osifp: (dc76e540) os_if: (d7b88000)

Aug 14 11:01:33  KERNEL(Birmingham-AP06-23364@192.168.12.17): [486986.339533] VAP device aruba001 created osifp: (d9562540) os_if: (dcca0000)

Aug 14 11:01:33  KERNEL(Birmingham-AP03-23367@192.168.12.18): [486987.684800] VAP device aruba101 created osifp: (dc768540) os_if: (d6b58000)

Aug 14 11:01:33  KERNEL(Birmingham-AP07-23365@192.168.12.13): [486985.257434] VAP device aruba000 created osifp: (da637540) os_if: (da188000)

Aug 14 11:01:33  KERNEL(Birmingham-AP07-23365@192.168.12.13): [486985.310917] VAP device aruba001 created osifp: (da635540) os_if: (da388000)

Aug 14 11:01:34  KERNEL(Birmingham-AP06-23364@192.168.12.17): [486987.333754] VAP device aruba100 created osifp: (d9cae540) os_if: (da0d0000)

Aug 14 11:01:34  KERNEL(Birmingham-AP06-23364@192.168.12.17): [486987.388768] VAP device aruba101 created osifp: (d9cac540) os_if: (dc180000)

Aug 14 11:01:34  KERNEL(Birmingham-AP07-23365@192.168.12.13): [486986.304950] VAP device aruba100 created osifp: (dd1a3540) os_if: (db010000)

Aug 14 11:01:34  KERNEL(Birmingham-AP07-23365@192.168.12.13): [486986.359839] VAP device aruba101 created osifp: (dd1a1540) os_if: (d9738000)

Aug 14 11:01:46 :399838:  <3556> <WARN> |stm|  Resource 'Total APs' has exceeded  80%  threshold (actual:90%).

Highlighted
Guru Elite

Re: Random Drops For Whole Site

Just the controller, for now.

 

I would type "show log system 50" and see if there are any AP bootstraps.  If those APs are at the same site, and they have bootstraps, you have a local issue.

 

 


*Answers and views expressed by me on this forum are my own and not necessarily the position of Aruba Networks or Hewlett Packard Enterprise.*
ArubaOS 8.4 User Guide
InstantOS 8.3 User Guide
Airheads Knowledgebase
Airheads Learning Videos
Aruba Central Documentation
Sign up for Security Alerts
Aruba Technical Webinars
MVP

Re: Random Drops For Whole Site

OK i have cleared the master-local stats

 

I did a show log system 50 but no AP boot straps

 

It is all the Aps and controllers on one site however that have the issue - happened again at 16:35 last night

MVP

Re: Random Drops For Whole Site

Aug 14 16:34:57  KERNEL(Birmingham-AP08-23362@192.168.12.15): [503390.176022] Warning, Received Frags in BA session from (8c:85:90:8c:77:5b)
Aug 14 16:34:57  KERNEL(Birmingham-AP08-23362@192.168.12.15): [503390.176022] Dropping frames because is not allowed by 802.11
Aug 14 16:34:57  KERNEL(Birmingham-AP08-23362@192.168.12.15): [503390.178740] Warning, Received Frags in BA session from (8c:85:90:8c:77:5b)
Aug 14 16:34:57  KERNEL(Birmingham-AP08-23362@192.168.12.15): [503390.178771] Dropping frames because is not allowed by 802.11
Aug 14 16:34:57  KERNEL(Birmingham-AP08-23362@192.168.12.15): [503390.180021] Warning, Received Frags in BA session from (8c:85:90:8c:77:5b)
Aug 14 16:34:57  KERNEL(Birmingham-AP08-23362@192.168.12.15): [503390.180021] Dropping frames because is not allowed by 802.11
Aug 14 16:35:21  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap
Aug 14 16:35:29  KERNEL(Birmingham-AP07-23365@192.168.12.13): [503421.509012] VAP device aruba000 created osifp: (db952540) os_if: (da300000)
Aug 14 16:35:29  KERNEL(Birmingham-AP07-23365@192.168.12.13): [503421.563870] VAP device aruba001 created osifp: (db950540) os_if: (d9be8000)
Aug 14 16:35:30  KERNEL(Birmingham-AP04-23368@192.168.12.14): [503425.154779] VAP device aruba000 created osifp: (dc73f540) os_if: (dbe68000)
Aug 14 16:35:30  KERNEL(Birmingham-AP07-23365@192.168.12.13): [503422.557934] VAP device aruba100 created osifp: (d9f5b540) os_if: (d9738000)
Aug 14 16:35:30  KERNEL(Birmingham-AP04-23368@192.168.12.14): [503425.208418] VAP device aruba001 created osifp: (dc73d540) os_if: (d7c80000)
Aug 14 16:35:30  KERNEL(Birmingham-AP07-23365@192.168.12.13): [503422.612792] VAP device aruba101 created osifp: (d9f5e540) os_if: (da998000)
Aug 14 16:35:31  KERNEL(Birmingham-AP04-23368@192.168.12.14): [503425.302483] VAP device aruba100 created osifp: (da93c540) os_if: (da9c0000)
Aug 14 16:35:31  KERNEL(Birmingham-AP04-23368@192.168.12.14): [503425.357372] VAP device aruba101 created osifp: (da938540) os_if: (dd148000)
Aug 14 16:35:31  KERNEL(Birmingham-AP06-23364@192.168.12.17): [503424.485457] VAP device aruba000 created osifp: (dc48c540) os_if: (dc8e0000)
Aug 14 16:35:31  KERNEL(Birmingham-AP06-23364@192.168.12.17): [503424.541033] VAP device aruba001 created osifp: (dc48e540) os_if: (da200000)
Aug 14 16:35:32  fpapps[3457]: <399838> <3460> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap mapid 0x44a1, vlanid 9, flags = 0x2 uplink_priority 0
Aug 14 16:35:32  KERNEL(Birmingham-AP06-23364@192.168.12.17): [503425.534723] VAP device aruba100 created osifp: (daa92540) os_if: (da300000)
Aug 14 16:35:32  KERNEL(Birmingham-AP06-23364@192.168.12.17): [503425.589643] VAP device aruba101 created osifp: (daa96540) os_if: (d84e8000)
Aug 14 16:35:39  cfgm[3433]: <399816> <3433> <ERRS> |cfgm|  handle_read: State(READY:UPDATE SUCCESSFUL:CFGID-271:PEND-269:INITCFGID:265) FD=27:Failure receiving heartbeat response header information Result=-1 Err=Connection reset by peer
Aug 14 16:35:39  cfgm[3433]: <399816> <3433> <ERRS> |cfgm|  Starting BOC log collection timer
Aug 14 16:35:49  KERNEL(Birmingham-AP03-23367@192.168.12.18): [503443.297734] VAP device aruba000 created osifp: (da9bc540) os_if: (d9d68000)
Aug 14 16:35:49  KERNEL(Birmingham-AP03-23367@192.168.12.18): [503443.353342] VAP device aruba001 created osifp: (da9bd540) os_if: (d9e68000)
Aug 14 16:35:50  KERNEL(Birmingham-AP03-23367@192.168.12.18): [503444.346688] VAP device aruba100 created osifp: (dc768540) os_if: (d9c20000)
Aug 14 16:35:50  KERNEL(Birmingham-AP03-23367@192.168.12.18): [503444.401764] VAP device aruba101 created osifp: (dc76c540) os_if: (dcfa0000)
Aug 14 16:40:13  KERNEL(Birmingham-AP09-24120@192.168.12.6): [503706.035285] Warning, Received Frags in BA session from (f0:18:98:06:10:e5)
Aug 14 16:40:13  KERNEL(Birmingham-AP09-24120@192.168.12.6): [503706.035285] Dropping frames because is not allowed by 802.11
Aug 14 16:40:13  KERNEL(Birmingham-AP09-24120@192.168.12.6): [503706.035410] Warning, Received Frags in BA session from (f0:18:98:06:10:e5)
Aug 14 16:40:13  KERNEL(Birmingham-AP09-24120@192.168.12.6): [503706.035441] Dropping frames because is not allowed by 802.11
Aug 14 16:56:21  KERNEL(Birmingham-AP09-24120@192.168.12.6): [504674.104170] Warning, Received Frags in BA session from (8c:85:90:62:16:ef)
Aug 14 16:56:21  KERNEL(Birmingham-AP09-24120@192.168.12.6): [504674.104201] Dropping frames because is not allowed by 802.11
Aug 14 16:56:21  KERNEL(Birmingham-AP09-24120@192.168.12.6): [504674.104295] Warning, Received Frags in BA session from (8c:85:90:62:16:ef)
Aug 14 16:56:21  KERNEL(Birmingham-AP09-24120@192.168.12.6): [504674.104295] Dropping frames because is not allowed by 802.11
Guru Elite

Re: Random Drops For Whole Site

Is this a Virtual Machine?  I also saw a log before that said you are running this controller at 90% capacity, which is not advised (EDIT: 80% is the typical advice in a large network).

 

If this is important and you want this resolved quickly, you should open a TAC case.  There could be a number of things at play, here.


*Answers and views expressed by me on this forum are my own and not necessarily the position of Aruba Networks or Hewlett Packard Enterprise.*
ArubaOS 8.4 User Guide
InstantOS 8.3 User Guide
Airheads Knowledgebase
Airheads Learning Videos
Aruba Central Documentation
Sign up for Security Alerts
Aruba Technical Webinars
MVP

Re: Random Drops For Whole Site

Mobility Master is running on Virtual Machines


The Mobility Devices are on site and 7005 Aruba Controllers 2 of them terminatiing only 10 APS and also around 200 clients - should be sufficient from my calculations 


I think the usage you are referring to is the 90% AP usage set in the threshold list: Aug 14 11:01:46 :399838:  <3556> <WARN> |stm|  Resource 'Total APs' has exceeded  80%  threshold (actual:90%).


How do I check the usage or capacity of the device?

Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: