Wireless Access

last person joined: yesterday 

Access network design for branch, remote, outdoor and campus locations with Aruba access points, and mobility controllers.
Expand all | Collapse all

Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

  • 1.  Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 01, 2018 11:00 AM

    Anyone else since a cluster not come up in L2 connected status after verifing network vlan tags to the controllers, vlans on the controller, and noticing MCs are mismatching on non-exisiting vlans? Currently have a TAC case open...

     

     

    Also any heads up on OS 8.3 or 8.2.1 release?



  • 2.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 01, 2018 12:40 PM
    Do you have any additional VLANs configured for each controller ?



    Thank you

    Pardon typos sent from Mobile


  • 3.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 01, 2018 12:51 PM

    nope the vlans that it is mismatching on do not exisit anywhere in the configuration of either controller. TAC CERT guys is also stumped on the issue.



  • 4.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 01, 2018 01:35 PM

    Check which vlans are not properly working on the vlan-probe status.

     

    I think the command is "show lc-cluster vlan-probe status", I had to remove vlan 1 from the configuration to make it a L2 cluster instead of a L3.

     

    The command for that is 'lc-cluster exclude-vlan "1"'

     

    Do that on each MD that is configured inside the cluster. Also, I suggest you follow the configuration on the Aruba Solution Exchange (https://ase.arubanetworks.com/).

     

    Hope you find the problem!

     

    Cheers.



  • 5.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 01, 2018 05:25 PM
    Also, if you are using a native vlan on the upstream trunk, try excluding that as well.


    #AirheadsMobile


  • 6.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 01, 2018 09:16 PM

    Alright so we figured it out. 

     

    1st there is a display bug on the MM when viewing the clusters. If a cluster is in L3 and has mismatch vlans and the vlan are atleast in my case 2790, 2989, 2990 the gui displays 2,790 / 2,989 / 2,990 making it look like 2 different vlans but possibly displaying the number like your tradition school 1,000 in math. This is confusing when in network you don't insert a comma.

     

    2nd the stack of cisco 3850 these controllers are attached to allowed us to create an ip interface for a vlan that didn't actually exisit. created the vlan and 2790 was up and working

     

    3rd i am in the process of moving configuration from a 6.4.4.16 code to new 8.2.0.2 by rebuilding the configuration to clean it up. By accident vlan 2989-2990 made it into our new 8.2.0.2 instance when it shouldn't have. deleted those vlans and now l2 connected.



  • 7.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 02, 2018 03:33 AM

    Good troubleshooting and fixing it!

     

    On to the next.....



  • 8.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Jan 08, 2019 07:58 AM

    Hello Airheads,

    i have ArubaOS 8.3.x running (2x VMC on VLAN 81 and 2x MM on VLAN 80).
    My problem is that L2 cluster works fine as log as both VMCs running on the same VM-Host. If one is migrated to an other host, the L2 cluster is broken.

    But all maschines are always reachable by IP. And VRRP works also.

    Only the L2 Connection is gone.

     

    Must be something on the VM side, i think.

    Any suggestions?

     

    Thank you!



  • 9.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Jan 09, 2019 06:31 PM

    Hi,

     

    Please double check if all VLAN's on the cluster are 'connected' all the way to the other esx host. If you don't excluded VLAN's they must all be constitent between the complete setup.

     

    Hope this helps

     



  • 10.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 14, 2019 08:48 PM
    Hello "hno", I have exactly the same issue, if both vMCs were hosted in the same VMware host the cluster works as expected, if the vMCs were hosted in separated VMware hosts, the cluster not works, the status is two vMCs in Isolated Leader. All other communication between vMCs are ok, ping, arp, vrrp, etc. Only the cluster not come up. Please if you have some manner to solve it post here. Since for full HA the two vMCs should not be running in the same VMware host.


  • 11.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 15, 2019 01:36 AM
    Hello

    Did you check:

    1. Full L2 connectivity between the esx hosts
    2. Forged transmits and promiscuous mode both on on all esx’s

    Can you use a separate vswitch just for the vmc’s and vmm?



  • 12.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 15, 2019 07:03 AM
    Yes I already checked full L2 communication, vrrp between vmc are ok too. Promiscuous, forged transmits and mac changes are enabled, Net.ReversePathPromisc also enabled too. I will try to use separate vswitch. Now I am using the same distributed vswitch for vmc. Tks.


  • 13.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 15, 2019 11:35 AM

    What version of AOS on the VMC, and what version of ESX?



  • 14.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 15, 2019 12:05 PM

    ESX 6.5

    AOS 8.2.2.3 also tried 8.4.0.0 with same result.



  • 15.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 18, 2019 05:01 PM

    I can not create a separate vswitch for each vMC because I do not have uplink for it, I tried to put each vMC in a separated vPortGroup but the result is the same.

     

    Cluster Info Table
    ------------------

    Type IPv4 Address Priority Connection-Type STATUS
    ---- --------------- -------- --------------- ------
    self 172.26.1.101 128 N/A ISOLATED (Leader)
    peer 172.26.1.102 128 N/A SECURE-TUNNEL-ESTABLISHED
    (MC-01) #

     

     

    Cluster Info Table
    ------------------
    Type IPv4 Address Priority Connection-Type STATUS
    ---- --------------- -------- --------------- ------
    peer 172.26.1.101 128 N/A SECURE-TUNNEL-ESTABLISHED
    self 172.26.1.102 128 N/A ISOLATED (Leader)
    (MC-02) #

     

    Attached I put a capture from two vMCs, one thing I found that is not good is the fact of PAPI packets from one vMC are not arriving at the other one, another thing is that mac of PAPI packets have source/dest all 0s(00:00:00:00:00:00). The IP address information are correct, 802.1q information also are present in some ISAKMP packets, my management port is configured as access, in my mind this kind of packets should not have 802.1q info.

    Attachment(s)

    zip
    filter-2-2.zip   296K 1 version
    zip
    filter-1-2.zip   385K 1 version


  • 16.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 18, 2019 06:57 PM

    How are you checking full L2 connectivity between the two VMCs? 

     

    I'm trying to carve out the time to stand up a second ESX box in my lab to recreate, but have not encountered this problem in production networks.

     

    Can you add screen shots showing the VMC connectivity within the VM environment, as well as the interfaces as seen from AOS? The cluster profile would also be helpful, to verify any excluded VLANs listed there.



  • 17.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 20, 2019 03:02 PM

    Bellow you will see L2 full communication between MCs. In your lab do you have 2 ESX separeted hosts?

    icmp.PNGcluster-profile-1.PNGcluster-profile-2.PNGlc-cluster.PNG



  • 18.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 20, 2019 10:20 PM
    My topology have HP virtual connect, I am think about some issue in it.

    My Topology
    ESXI1---vSwitch---VirtualConnect1---DellSwitchToR1---DellSwitchToR2---VirtualConnect2---vSwitch---ESXI2

    Capture of packets made in the interfaces of Dell switches facing to VirtualConnect, I could see Aruba PAPI protocol from vMC1 and vMC2, in both directions, a capture made at vMC I saw only PAPI packets leaving but nothing arriving, I think VirtualConnect are discarding Aruba PAPI protocol for some reason, but ICMP packets and all other network protocol completes as expected. If both vMCs were hosted in the same ESXI(only for tests) the communication is made direct by vSwitch and all looks good, I am think that VirtualConnect have some issue to work with Aruba PAPI.

    Any one here with knowledge of HP VirtualConnect?


  • 19.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 24, 2019 09:57 PM

    What's the output of  

     

    show lc-cluster vlan-probe status

     

    Resolved my issue via the VLAN_FAIL column



  • 20.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 24, 2019 10:28 PM

    You can also compare the configurations between MDs from your MM.

     

    Here's an example in my lab.

    (PDC-MM1-2.2) [mynode] #show configuration node-hierarchy

    Default-node is not configured. Autopark is disabled.

    Configuration node hierarchy
    ----------------------------
    Config Node Type Name
    ----------- ---- ----
    / System
    /md System
    /md/testing123 Group
    /md/The-Beatles Group
    /md/The-Beatles/Cluster-1 Group
    /md/The-Beatles/Cluster-1/00:0c:29:c1:fc:36 Device SDC-VMC2-Cluster1
    /md/The-Beatles/Cluster-1/00:0c:29:dc:f4:ec Device PDC-VMC1-Cluster1
    /md/The-Beatles/Cluster-2 Group
    /md/The-Beatles/Cluster-2/00:0c:29:ec:5a:94 Device SDC-VMC2-Cluster2
    /md/The-Beatles/Cluster-2/00:0c:29:fb:7a:ae Device PDC-VMC1-Cluster2
    /mm System
    /mm/mynode System

     

    You can use the context attribute on the below command for additional clarity.
    (PDC-MM1-2.2) [mynode] #show configuration diff /md/The-Beatles/Cluster-1/00:0c:29:dc:f4:ec /md/The-Beatles/Cluster-1/00:0c:29:c1:fc:36
    interface vlan 60
    - ip address 10.60.2.11 255.255.240.0
    + ip address 10.60.2.12 255.255.240.0
    interface vlan 610
    - no suppress-arp
    - interface vlan 99
    - ip address dhcp-client
    vrrp 2
    - preempt
    - priority 200
    - mgmt-user admin root 26e4596d013ae252c871a9f605e475906ab8c4fc5841980704 max-concurrent-sessions 0
    + mgmt-user admin root 0db05bef01e2e5c7b59adc41bbb2ee858e0a8dbba25463a087 max-concurrent-sessions 0
    interface gigabitethernet 0/0/1
    interface gigabitethernet 0/0/2
    - switchport trunk allowed vlan 1,60
    + switchport trunk allowed vlan 1
    - trusted vlan 1,60
    + trusted vlan 1
    - hostname PDC-VMC1-Cluster1
    + hostname SDC-VMC2-Cluster1
    (PDC-MM1-2.2) [mynode] #

     



  • 21.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 09:14 AM

    The output of vlan-probe status looks good.

    (MC-01) [MDC] #show lc-cluster vlan-probe status

    Cluster VLAN Probe Status
    -------------------------
    Type IPv4 Address REQ-SENT REQ-FAIL ACK-SENT ACK-FAIL REQ-RCVD ACK-RCVD VLAN_FAIL CONN-TYPE START/STOP
    ---- --------------- -------- -------- -------- -------- -------- -------- --------- --------- ----------
    peer 172.26.1.102 0 0 0 0 0 0 0 N/A 0/ 49
    (MC-01) [MDC] #

     

    (MC-02) [MDC] #show lc-cluster vlan-probe status

    Cluster VLAN Probe Status
    -------------------------
    Type IPv4 Address REQ-SENT REQ-FAIL ACK-SENT ACK-FAIL REQ-RCVD ACK-RCVD VLAN_FAIL CONN-TYPE START/STOP
    ---- --------------- -------- -------- -------- -------- -------- -------- --------- --------- ----------
    peer 172.26.1.101 0 0 0 0 0 0 0 N/A 0/ 49
    (MC-02) [MDC] #

     

    About config diffrent the only different configs that I have are ip address, priority of vrrp and hostname.



  • 22.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 09:54 AM

    Ah I saw your previous screenshots. It appears its failing to establish the IPSec tunnel.

     

    The traffic needed between MCs =

    (IPSec Phase 1) ISAKMP UDP 500

    (IPSec Phase 2) ESP IP Protocol 50

     

    Truly make sure theres no fw or acls between.

     

    MC-01:show crpyto isakmp sa

    MC-01:show crypto ipsec sa

     

    Setting up debug to see what's failing:

    MM1 [md] (config) # change-config-node MC-01

    config t

    logging security process authmgr level debug

    logging security process crypto level debug

    logging system processs cluster_mgr level debugging

    wr me

     

    MC-01#  show log security 100

     

    Buried in here will tell us if its Phase 1 or Phase 2. 

     

     



  • 23.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 10:19 AM

    Crypto-SA-1.PNGCrypto-SA-2.PNG

    Hello Qlimax, as lc-cluster status suggest I can see that tunnel is stablished, in MC-01 I see tunnels with MM, AP and MC-02, in MC-02 I see tunnel with MM and MC-01. The output of logs you suggest to enable only show messages of rouge SSID detected, take a look:

     

    Mar 25 11:08:24 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID a2:2c:36:b3:7e:23 and SSID MirrorCAST 36B37E23 on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
    Mar 25 11:10:54 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID dc:bf:e9:3e:ab:0a and SSID Moto G (5S) 1017 on CHANNEL 4) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
    Mar 25 11:11:58 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID 30:cb:f8:1c:79:87 and SSID AndroidAP on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
    Mar 25 11:12:07 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID 38:9a:f6:84:30:69 and SSID AndroidAP3069 on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
    Mar 25 11:12:45 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID b0:6e:bf:d7:4d:09 and SSID Euzebio on CHANNEL 11) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
    Mar 25 11:13:19 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID 02:57:c1:ed:c7:89 and SSID tancredo on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
    Mar 25 11:13:31 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID fc:64:3a:b8:a2:58 and SSID AndroidAPA258 on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.



  • 24.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 10:26 AM

    Sorry, read through the history but didn't see note of it. You are seeing this when they are spread across two separate hypervisors but with the same dvSwitch correct? Fot the vSwitch in question, is it using two physical adapters on the vSwitch? If so, is LACP used between the hypervisors and the physical uplink switch or is it just using NIC Teaming?



  • 25.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 10:50 AM

    jhoward, I am using just one NIC for vswitch, do not have lacp nor nic team. When using same hypervisor everything looks good, with no change of AOS config, I have in my mind that issue is something about compatibility with VMware enviroment, maybe.



  • 26.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 11:39 AM

    Does your output still look the same in previous screenshots?

     

    show lc-cluster group-membership

     

    Are they stuck on isolated and "Secure tunnel negotiating"

     

    If so the traffic between the MCs is either being filtered somehow or there is a misconfig in the IPSec config I'd think.



  • 27.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 12:43 PM

    The role is stucked "Isolated (Leader)" and status keep moving between disconnected, negotiating and established. I do not have any filter in the path, just L2 network, vswitch, virtual connect switch and standard network switch, without any filter.



  • 28.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 02:08 PM
    Is phase 1 or 2 breaking

    Sent from my IPhone


  • 29.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 04:23 PM

    Seems that phase 1 and 2 are stablished.



  • 30.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 04:34 PM

    I dont believe so if they're stuck in the state of "Secure Tunnel Negotiating" 

    show lc-cluster group-membership

     

    Why it could be happening? Not sure...I have cluster members on different hosts. 



  • 31.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 25, 2019 04:59 PM

    It stuck in "Secure-Tunnel-Stablished" stay some time in it, since not receives PAPI packets it go back to disconnect, then negotiating and again to stablished. It not stuck in negotiating, sorry if I am not clear enough.



  • 32.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Mar 28, 2019 01:59 PM

    FYI I had another issue with having my MCs not forming an L2 after adding a few vlans. Come to find out I missed some switchport tagging on redundant links to my hosts. Just another thing to check....



  • 33.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Jun 19, 2019 02:09 PM

    Just to update the case, we solve the issue creating 2 another new ESXi servers and migrate MC to these new servers, as a magic everthing working as expected now. We couldn't identify the rason for this issue, but looks like something specific with hardware HP Blade System with Virtual Connect.



  • 34.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Jul 19, 2019 02:21 PM

    Hi,

     

    Please let us know the HPE Blade Server model number details.

     



  • 35.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Jul 19, 2019 03:35 PM

    HP BladeSystem C7000 witch Flex10 Module(VirtualConnect 4.63)

     

    TAC concludes that around 99% of papi packets was lost



  • 36.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Jul 19, 2019 04:11 PM

    Many of the blade systems with their integrated virtual network stack behind it don't usually support promiscuous mode or forged transmits, and require usually a direct VM to physical port map of switch port to the virtual interface of the VMM or VMC. Either that or they don't handle a virtualized mac on the VM that doesn't directly map to a vMAC of the virtual interface on the VM. 



  • 37.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Jul 22, 2019 10:07 AM

    Hello Jhoward, the problem is that in this case we do not have available interface to dedicate for MDs, all physical NICs are in use.



  • 38.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Jul 20, 2019 02:41 AM

    Hi

    Can you please update which HPE blade server was used ? CPU and memory and all details. 

    and which VMC SKUs were being used in cluster?



  • 39.  RE: Aruba OS 8.2.0.2 Cluster not coming up in L2 Connected

    Posted Jul 23, 2019 04:57 AM

    Hi Airheads,

     

    tomorrow we do a troubleshooting session with an Aruba WLAN- and an ESX Guy. 

    I'll keep you informed.

     

    best regards,

    Michael