Wireless Access

Reply
Highlighted
New Contributor

RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

Hi

 

I currently have the problem with RAP's that constantly rebuild the VPN connection to the controller cluster. The connection of CAP to the controller cluster works fine.

 

When the VPN connection is established everything looks good, the tunnel is established, it gets an Inner IP from the MM Cluster and the SSID is provided. After that the connection is immediately aborted and the VPN connection is reestablished.

 

In order not to work on the productive environment I have built a LAB environment with a 1to1 copy of the productive environment. With the following topology (The whole environment is running ArubaOS 8.5.0.5.):

rap_clu_enviro.png

 

On the RAP I have activated debug. Here is an section of the establishment and termination of the VPN connection:

 

[  334.349404] uol_add_tun_l3:211 tun add l3(1.1.1.10->172.36.201.108): mtu=1500, l2hlen=14, ifnum=0 odev=tun0

[  339.068550] aruba_usb_power_control:1153 en:1 status:1

[  339.202978] uol_update_mtu:336 tun update (1.1.1.10->172.36.201.108): mtu updated to 1200

[  340.272807] Tue Jan  7 16:25:05 2020:121       146.227098 wl0 set max_distance 0, radio type 0, outdoor_ap 0,

[  340.392665] Tue Jan  7 16:25:05 2020:122       146.346966 wl0 apply max distance 0(not mesh radio)

[  340.581142] Tue Jan  7 16:25:05 2020:123       146.535433 wl1 set max_distance 0, radio type 0, outdoor_ap 0,

[  340.700997] Tue Jan  7 16:25:05 2020:124       146.655297 wl1 apply max distance 0(not mesh radio)

[  341.881957] wl0: wlc_ampdu_tx_set:  AGG Mode = HOST txmaxpkts 6

[  341.952879] wl0: wlc_ampdu_tx_set: 0x3, err 0

[  342.008550] Tue Jan  7 16:25:07 2020:vap_device_event: dev aruba000 (0:0) has been registered

[  342.110759] asap_vap_attach: asap_vaps[0][0] attached, vap: d8feec00, dev: aruba000 dc6b5000

[  342.415920] Tue Jan  7 16:25:07 2020:125       148.370198

[  342.415924]  first_wmm_ac_dscp[56] = c8

[  342.525344] Tue Jan  7 16:25:07 2020:126       148.479646

[  342.525348]  wmm_ac_dscp_map[56] = 0/

[  342.637812] Tue Jan  7 16:25:07 2020:127       148.592104

[  342.637816]  first_wmm_ac_dscp[57] = c8

[  342.749336] Tue Jan  7 16:25:07 2020:128       148.703637

[  342.749340]  wmm_ac_dscp_map[57] = 0/

[  342.861850] Tue Jan  7 16:25:07 2020:129       148.816147

[  342.861855]  first_wmm_ac_dscp[58] = c8

[  342.973373] Tue Jan  7 16:25:08 2020:130       148.927672

[  342.973377]  wmm_ac_dscp_map[58] = 0/

[  343.085774] Tue Jan  7 16:25:08 2020:131       149.040069

[  343.085778]  first_wmm_ac_dscp[59] = c8

[  343.197299] Tue Jan  7 16:25:08 2020:132       149.151601

[  343.197303]  wmm_ac_dscp_map[59] = 0/

[  343.313679] Tue Jan  7 16:25:08 2020:133       149.267969 aruba000: Scheduling VAP DOWN CONFIG update

[  343.426354] Tue Jan  7 16:25:08 2020:134       149.380652 aruba000: Forwarding Mode set to 0

[  343.527451] Tue Jan  7 16:25:08 2020:135       149.481751 aruba000: ESSID set to BlackTest

[  343.626452] Tue Jan  7 16:25:08 2020:136       149.580753 aruba000: WME_APSD set to 1

[  343.720315] Tue Jan  7 16:25:08 2020:137       149.674613 aruba000: WME set to 1

[  343.808980] Tue Jan  7 16:25:08 2020:138       149.763275 aruba000: Basic Rate set to 336

[  343.906940] Tue Jan  7 16:25:08 2020:139       149.861241 aruba000: Supported Rate set to 4080

[  344.010107] Tue Jan  7 16:25:09 2020:140       149.964408 aruba000: Supported VHT mcsmap set to 0xaaaa

[  344.121620] Tue Jan  7 16:25:09 2020:141       150.075920 aruba000: Beacon rate set to 65535

[  344.222729] wl0: wlc_ampdu_tx_set: AGG Mode = MAC+AQM txmaxpkts 1024

[  344.298802] wl0: wlc_ampdu_tx_set: 0x3, err 0

[  344.350919] Rx-ampdu enable/disable is not supported for this AP type

[  344.428093] Tue Jan  7 16:25:09 2020:142       150.382389 aruba000: HT Mode set to 1

[  344.520842] Tue Jan  7 16:25:09 2020:143       150.475145 aruba000: 40MHz Enabled

[  344.610517] Tue Jan  7 16:25:09 2020:144       150.564816 aruba000: VHT Mode set to 1

[  344.704363] Tue Jan  7 16:25:09 2020:145       150.658661 aruba000: 80MHz Enabled

[  344.793995] Tue Jan  7 16:25:09 2020:146       150.748296 aruba000: VAP DOWN CONFIG DONE

[  344.923219] Tue Jan  7 16:25:09 2020:147       150.877511 aruba000: Scheduling VAP UP CONFIG update

[  345.031690] Tue Jan  7 16:25:10 2020:148       150.985989 aruba000: MCAST_RATE_OPT set to 0

[  345.131792] Tue Jan  7 16:25:10 2020:149       151.086090 aruba000: STA Ageout set to 1000

[  345.230849] Tue Jan  7 16:25:10 2020:150       151.185150 aruba000: Max retries set to 8

[  345.327782] Tue Jan  7 16:25:10 2020:151       151.282082 aruba000: RTS Threshold set to 2333

[  345.430114] Tue Jan  7 16:25:10 2020:152       151.384402 aruba000: A-MPDU: Enabled, Tx Max 65535 Rx Max 3 Min Spacing 5

[  345.560476] Tue Jan  7 16:25:10 2020:153       151.514771 aruba000: VHT MPDU-size set to 1

[  345.659692] Tue Jan  7 16:25:10 2020:154       151.613989 aruba000: RX AMSDU Enable

[  345.751410] Tue Jan  7 16:25:10 2020:155       151.705709 aruba000: TX AMSDU: count be set to 2, count bk set to 2, count vi set to 2, count vo set to 0

[  345.914997] Tue Jan  7 16:25:10 2020:156       151.869296 aruba000: 20MHz GI Enabled, 40MHz GI Enabled, 80MHz GI Enabled

[  346.045246] Tue Jan  7 16:25:11 2020:157       151.999547 aruba000: LDPC Enabled

[  346.133951] Tue Jan  7 16:25:11 2020:158       152.088244 aruba000: STBC TX Enabled

[  346.225846] Tue Jan  7 16:25:11 2020:159       152.180140 aruba000: STBC RX Enabled

[  346.317561] Tue Jan  7 16:25:11 2020:160       152.271863 aruba000: legacy-allowed Enabled

[  346.416619] Tue Jan  7 16:25:11 2020:161       152.370917 aruba000: TXBF Enabled

[  346.505248] Tue Jan  7 16:25:11 2020:162       152.459548 aruba000: Max Tx Failures 0

[  346.600129] Tue Jan  7 16:25:11 2020:163       152.554429 aruba000: vap max-clients to 64

[  346.698275] wl0: wlc_del_ie: IE not in list

[  346.748326] wl0: wlc_iovar_op: BCME -30 (Not Found)

[  346.806683] wl0: set ap location - coordinates are not provisioned

[  346.880812] Tue Jan  7 16:25:11 2020:164       152.835109 aruba000: Multicast rate set to 65535 (65535)

[  346.993412] Tue Jan  7 16:25:12 2020:165       152.947710 aruba000: Refresh Direction set to 2

[  347.096676] Tue Jan  7 16:25:12 2020:166       153.050968 aruba000: VAP UP CONFIG Update DONE

[  347.199312] Tue Jan  7 16:25:12 2020: vap_device_event: dev aruba000 (0:0) is up

[  347.354741] wl1: wlc_ampdu_tx_set:  AGG Mode = HOST txmaxpkts 6

[  347.425634] wl1: wlc_ampdu_tx_set: 0x3, err 0

[  347.481112] Tue Jan  7 16:25:12 2020:vap_device_event: dev aruba100 (1:0) has been registered

[  347.583314] asap_vap_attach: asap_vaps[1][0] attached, vap: d8fc1c00, dev: aruba100 d84f6800

[  347.890374] Tue Jan  7 16:25:12 2020:167       153.844668

[  347.890379]  first_wmm_ac_dscp[56] = c8

[  347.999819] Tue Jan  7 16:25:13 2020:168       153.954119

[  347.999823]  wmm_ac_dscp_map[56] = 0/

[  348.111266] Tue Jan  7 16:25:13 2020:169       154.065560

[  348.111271]  first_wmm_ac_dscp[57] = c8

[  348.222808] Tue Jan  7 16:25:13 2020:170       154.177102

[  348.222812]  wmm_ac_dscp_map[57] = 0/

[  348.333623] Tue Jan  7 16:25:13 2020:171       154.287918

[  348.333628]  first_wmm_ac_dscp[58] = c8

[  348.445147] Tue Jan  7 16:25:13 2020:172       154.399449

[  348.445151]  wmm_ac_dscp_map[58] = 0/

[  348.559481] Tue Jan  7 16:25:13 2020:173       154.513771

[  348.559487]  first_wmm_ac_dscp[59] = c8

[  348.671009] Tue Jan  7 16:25:13 2020:174       154.625311

[  348.671013]  wmm_ac_dscp_map[59] = 0/

[  348.787541] Tue Jan  7 16:25:13 2020:175       154.741833 aruba100: Scheduling VAP DOWN CONFIG update

[  348.900235] Tue Jan  7 16:25:13 2020:176       154.854532 aruba100: Forwarding Mode set to 0

[  349.001347] Tue Jan  7 16:25:14 2020:177       154.955645 aruba100: ESSID set to BlackTest

[  349.100370] Tue Jan  7 16:25:14 2020:178       155.054667 aruba100: WME_APSD set to 1

[  349.194268] Tue Jan  7 16:25:14 2020:179       155.148565 aruba100: WME set to 1

[  349.282968] Tue Jan  7 16:25:14 2020:180       155.237260 aruba100: Basic Rate set to 3

[  349.378865] Tue Jan  7 16:25:14 2020:181       155.333164 aruba100: Supported Rate set to 4095

[  349.482069] Tue Jan  7 16:25:14 2020:182       155.436362 aruba100: Supported VHT mcsmap set to 0x0

[  349.590486] Tue Jan  7 16:25:14 2020:183       155.544784 aruba100: Beacon rate set to 65535

[  349.691616] wl1: wlc_ampdu_tx_set: AGG Mode = MAC+AQM txmaxpkts 1024

[  349.767725] wl1: wlc_ampdu_tx_set: 0x3, err 0

[  349.819924] Rx-ampdu enable/disable is not supported for this AP type

[  349.897149] Tue Jan  7 16:25:14 2020:184       155.851445 aruba100: HT Mode set to 1

[  349.989918] Tue Jan  7 16:25:15 2020:185       155.944215 aruba100: 40MHz Disabled

[  350.080618] Tue Jan  7 16:25:15 2020:186       156.034915 aruba100: VAP DOWN CONFIG DONE

[  350.177890] wl1 bmac set short slot 1, wl status 1

[  350.267237] Tue Jan  7 16:25:15 2020:187       156.221529 aruba100: Scheduling VAP UP CONFIG update

[  350.375747] Tue Jan  7 16:25:15 2020:188       156.330040 aruba100: MCAST_RATE_OPT set to 0

[  350.475868] Tue Jan  7 16:25:15 2020:189       156.430164 aruba100: STA Ageout set to 1000

[  350.574954] Tue Jan  7 16:25:15 2020:190       156.529250 aruba100: Max retries set to 8

[  350.671983] Tue Jan  7 16:25:15 2020:191       156.626278 aruba100: Short Preamble set to 1

[  350.772060] Tue Jan  7 16:25:15 2020:192       156.726356 aruba100: RTS Threshold set to 2333

[  350.874415] Tue Jan  7 16:25:15 2020:193       156.828706 aruba100: A-MPDU: Enabled, Tx Max 65535 Rx Max 3 Min Spacing 5

[  351.004818] Tue Jan  7 16:25:16 2020:194       156.959109 aruba100: VHT MPDU-size set to 1

[  351.103960] Tue Jan  7 16:25:16 2020:195       157.058258 aruba100: RX AMSDU Enable

[  351.195683] Tue Jan  7 16:25:16 2020:196       157.149979 aruba100: TX AMSDU: count be set to 1, count bk set to 0, count vi set to 0, count vo set to 0

[  351.359308] Tue Jan  7 16:25:16 2020:197       157.313602 aruba100: 20MHz GI Enabled, 40MHz GI Enabled, 80MHz GI Disabled

[  351.490612] Tue Jan  7 16:25:16 2020:198       157.444916 aruba100: LDPC Enabled

[  351.579299] Tue Jan  7 16:25:16 2020:199       157.533592 aruba100: STBC TX Enabled

[  351.671235] Tue Jan  7 16:25:16 2020:200       157.625522 aruba100: STBC RX Enabled

[  351.762967] Tue Jan  7 16:25:16 2020:201       157.717270 aruba100: legacy-allowed Enabled

[  351.862052] Tue Jan  7 16:25:16 2020:202       157.816348 aruba100: Max Tx Failures 0

[  351.957092] Tue Jan  7 16:25:17 2020:203       157.911377 aruba100: vap max-clients to 64

[  352.055226] wl1: wlc_del_ie: IE not in list

[  352.105289] wl1: wlc_iovar_op: BCME -30 (Not Found)

[  352.163663] wl1: set ap location - coordinates are not provisioned

[  352.237836] Tue Jan  7 16:25:17 2020:204       158.192129 aruba100: Multicast rate set to 65535 (65535)

[  352.350454] Tue Jan  7 16:25:17 2020:205       158.304747 aruba100: Refresh Direction set to 2

[  352.453694] Tue Jan  7 16:25:17 2020:206       158.407992 aruba100: VAP UP CONFIG Update DONE

[  352.556428] Tue Jan  7 16:25:17 2020: vap_device_event: dev aruba100 (1:0) is up

[  352.651255] wl0: 2327 rx fifo 0 overflows!

[  361.765933] Tue Jan  7 16:25:26 2020:207       167.720224 aruba000: Scheduling VAP UP CONFIG update

[  361.874330] wl0: set ap location - coordinates are not provisioned

[  361.948603] Tue Jan  7 16:25:27 2020:208       167.902897 aruba000: VAP UP CONFIG Update DONE

[  362.051645] Tue Jan  7 16:25:27 2020:209       168.005935 aruba100: Scheduling VAP UP CONFIG update

[  362.160075] wl1: set ap location - coordinates are not provisioned

[  362.234339] Tue Jan  7 16:25:27 2020:210       168.188632 aruba100: VAP UP CONFIG Update DONE

[  362.911176] uol_del_tun:264 tun del (1.1.1.10->172.36.201.108)

[  362.982806] Cleanup bucketmap for essid BlackTest(0)

[  363.114723] Cleanup bucketmap for essid BlackTest(0)

[  364.314857] asap_vap_gone: vap gone notification for 0:0 dev dc6b5000 fw_mode 0

[  364.402431] asap_vap_gone: asap_vaps[0][0] gone, vap: d8feec00 dev: aruba000 dc6b5000

[  364.498623] wl0: interface DOWN for no vaps

[  364.548712] Tue Jan  7 16:25:29 2020: vap_device_event: dev aruba000 (0:0) is going down

[  364.652067] Tue Jan  7 16:25:29 2020: vap_device_event: dev aruba000 (0:0) is down

[  364.751460] Tue Jan  7 16:25:29 2020:vap_device_event: dev aruba000 (0:0) has been unregistered

[  364.857738] asap_vap_gone: vap gone notification for 1:0 dev d84f6800 fw_mode 0

[  364.945360] asap_vap_gone: asap_vaps[1][0] gone, vap: d8fc1c00 dev: aruba100 d84f6800

[  365.050475] wl1: interface DOWN for no vaps

[  365.100573] Tue Jan  7 16:25:30 2020: vap_device_event: dev aruba100 (1:0) is going down

[  365.202758] Tue Jan  7 16:25:30 2020: vap_device_event: dev aruba100 (1:0) is down

[  365.297952] Tue Jan  7 16:25:30 2020:vap_device_event: dev aruba100 (1:0) has been unregistered

 

Does anyone know the problem and has a solution for it? I am in contact with the TAC Support but not yet solved.

 

regards,

Christoph

 

Highlighted
Moderator

Re: RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

hi Christoph

if you have CLI access to the rap, please check the files
/tmp/rapper.txt
/tmp/sapd_debug_log.txt

these may have some better clue about what happened

-jeff

Highlighted
New Contributor

Re: RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

I am having the exact same issue, i am running 8.5.0.6. The RAP terminates on a 2 node cluster.The RAP device does not seem to be rebooting, it seems like the VPN tunnel keeps going down and up. I see the SSID briefly broadcasted and then it goes away.

 

Have you found a solution to your problem?

Highlighted
New Contributor

Re: RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

I just got off the phone with aruba tech, they told me that this was a reported bug in 8.5.0.5 and they believe it is also a but in 8.5.0.6. Were you able to find a AOS8 release that did not have this bug?

Highlighted
Occasional Contributor II

Re: RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

Did you get a chance to test it with 8.5.0.7? The tunnel seems to be stable, AP-315 and AP-515 seem to work fine, but I am hitting some huge performance issues with 303h.

Highlighted
New Contributor

Re: RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

I did not get a chance to test with 8.5.0.7, I downgraded to 8.5.0.4 and that resolved the issue for me. After downgrading the vpn tunnel from the RAP to the cluster remained stable. 

Highlighted
MVP Expert

Re: RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

Ivec and Christoph, if you can post your TAC case number and/or the bug ID that TAC cited, I can look in to it.


Jerrod Howard
Distinguished Technologist, TME
Highlighted
New Contributor

Re: RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

I had a TAC case open with Aruba for over two months. But a definitive solution was not found. The support was generally not very helpful, information in the emails was not read correctly. The support always performed the same tests with the RAP's. Even if it was the same technician.

The case will be handled in the next weeks together with an Aruba Presales technician.

Last update:
The RAP connection to the Mobility Cluster can be established as long as the RAP Public IP's are not defined in the cluster configuration. As soon as the RAP Public IP's are stored, the connection from the RAP to the controller is constantly re-established.

I could reproduce this problem with the following OS versions: 8.5.0.4; 8.5.0.5; 8.6.0.1; 8.6.0.2

 

Christoph

Highlighted
Moderator

Re: RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

hi Christoph

in your reproduction diagram, can I query a few things


a) Where are IPs 46.28.25.95/96 configured (other than as cluster rap IPs) ?

b) The RAPs are set for 46.28.25.x as master IP ?

 

c) do you have any LMS IPs configured in the ap system profile ?

 

 

Highlighted
Moderator

Re: RAP's constantly rebuild the VPN connection to the controller cluster (AOS 8.5.0.5)

FWIW, I set this up and tested on 8.5.0.6 with attached diagram, 3 RAPs up and stable for 12 hours.

 

(sg-7005-b) #show lc-cluster group-profile lab-rap-cluster

IPv4 Cluster Members
--------------------
CONTROLLER-IP  PRIORITY  MCAST-VLAN  VRRP-IP        VRRP-VLAN  GROUP-ID  RAP-PUBLIC-IP
-------------  --------  ----------  -------        ---------  --------  -------------
192.168.1.143  128       0           192.168.1.146  1          0         172.35.0.138
192.168.1.144  128       0           192.168.1.147  1          0         172.35.0.139

 

AP uptime is 12+ hours

(sg-7005-b) #show ap database long

AP Database
-----------
Name     Group  AP Type  IP Address  Status          Flags  Switch IP      Standby IP     Wired MAC Address  Serial #    Port  FQLN  Outer IP     User
----     -----  -------  ----------  ------          -----  ---------      ----------     -----------------  --------    ----  ----  --------     ----
rap303h  RAP    303H     10.10.10.1  Up 13h:6m:49s   Rc2    192.168.1.144  192.168.1.143  20:4c:03:28:05:10  CNFGK2R4JN  N/A   N/A   172.35.0.10  
rap345   RAP    345      10.10.10.3  Up 12h:51m:46s  Rc2    192.168.1.144  192.168.1.143  38:17:c3:c0:c2:ce  CNFJK5103T  N/A   N/A   172.35.0.10  
rap515   RAP    515      10.10.10.2  Up 12h:55m:12s  Rc2Sf  192.168.1.143  192.168.1.144  9c:8c:d8:c9:55:ac  CNH7KD52CY  N/A   N/A   172.35.0.10  

 

BSS uptime is uninterrupted for 12+ hours (e.g. no rebootstraps)

Aruba AP BSS Table
------------------
bss                ess          port  ip          phy    type  ch/EIRP/max-EIRP  cur-cl  ap name  in-t(s)  tot-t        mtu   acl-state  acl  fm  flags  cluster  datazone
---                ---          ----  --          ---    ----  ----------------  ------  -------  -------  -----        ---   ---------  ---  --  -----  -------  --------
9c:8c:d8:15:5a:d0  rap-cluster  N/A   10.10.10.2  a-HE  ap    153-/21.0/30.0    0       rap515   0        12h:45m:45s  1300  -          84   T          A        no
38:17:c3:8c:2c:f0  rap-cluster  N/A   10.10.10.3  a-VHT  ap    161-/21.0/29.7    0       rap345   0        12h:45m:57s  1300  -          84   T          A        no
38:17:c3:36:6b:30  rap-cluster  N/A   10.10.10.1  a-VHT  ap    120-/21.0/28.6    0       rap303h  0        12h:45m:57s  1300  -          84   T          A        no

 APs are talking to "public" IPs of the border f/w, as defined in the cluster profile

(sg-7005-b) #show ap remote debug nodelist history ap-name "rap345" 

Nodelist History
----------------
PID and Time                Node From        Role  Msg Type     Gen Num  Cluster  S-AAC
------------                ---------        ----  --------     -------  -------  -----
[ 3260]2020-03-29 21:57:42  192.168.1.144          ACT-NODELIST 8        Enabled  172.35.0.138
       Node List: 172.35.0.139        172.35.0.138        
(sg-7005-b) #
(sg-7005-b) #
(sg-7005-b) #show ap remote debug nodelist history ap-name "rap303h" 

Nodelist History
----------------
PID and Time                Node From        Role  Msg Type     Gen Num  Cluster  S-AAC
------------                ---------        ----  --------     -------  -------  -----
[ 4350]2020-03-29 21:44:44  192.168.1.144          ACT-NODELIST 8        Enabled  172.35.0.138
       Node List: 172.35.0.139        172.35.0.138        
(sg-7005-b) #
(sg-7005-b) #
(sg-7005-b) #show ap remote debug nodelist history ap-name "rap515" 

Nodelist History
----------------
PID and Time                Node From        Role  Msg Type     Gen Num  Cluster  S-AAC
------------                ---------        ----  --------     -------  -------  -----
[ 3272]2020-03-29 21:54:15  192.168.1.143          ACT-NODELIST 7        Enabled  172.35.0.139
       Node List: 172.35.0.138        172.35.0.139        
(sg-7005-b) #
(sg-7005-b) #

@Christoph, I'm happy to review your setup if you want, else, you can work through it with the Aruba SE this week. I'd be looking firstly at what is being done with 46.28.25.95/96 in your lab network.

Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: