Wireless Access

Anyone else since a cluster not come up in L2 connected status after verifing network vlan tags to the controllers, vlans on the controller, and noticing MCs are mismatching on non-exisiting vlans? Currently have a TAC case open...

 

 

Also any heads up on OS 8.3 or 8.2.1 release?

Do you have any additional VLANs configured for each controller ?



Thank you

Pardon typos sent from Mobile

nope the vlans that it is mismatching on do not exisit anywhere in the configuration of either controller. TAC CERT guys is also stumped on the issue.

Check which vlans are not properly working on the vlan-probe status.

 

I think the command is "show lc-cluster vlan-probe status", I had to remove vlan 1 from the configuration to make it a L2 cluster instead of a L3.

 

The command for that is 'lc-cluster exclude-vlan "1"'

 

Do that on each MD that is configured inside the cluster. Also, I suggest you follow the configuration on the Aruba Solution Exchange (https://ase.arubanetworks.com/).

 

Hope you find the problem!

 

Cheers.

Also, if you are using a native vlan on the upstream trunk, try excluding that as well.


#AirheadsMobile

Alright so we figured it out. 

 

1st there is a display bug on the MM when viewing the clusters. If a cluster is in L3 and has mismatch vlans and the vlan are atleast in my case 2790, 2989, 2990 the gui displays 2,790 / 2,989 / 2,990 making it look like 2 different vlans but possibly displaying the number like your tradition school 1,000 in math. This is confusing when in network you don't insert a comma.

 

2nd the stack of cisco 3850 these controllers are attached to allowed us to create an ip interface for a vlan that didn't actually exisit. created the vlan and 2790 was up and working

 

3rd i am in the process of moving configuration from a 6.4.4.16 code to new 8.2.0.2 by rebuilding the configuration to clean it up. By accident vlan 2989-2990 made it into our new 8.2.0.2 instance when it shouldn't have. deleted those vlans and now l2 connected.

Good troubleshooting and fixing it!

 

On to the next.....

Hello Airheads,

i have ArubaOS 8.3.x running (2x VMC on VLAN 81 and 2x MM on VLAN 80).
My problem is that L2 cluster works fine as log as both VMCs running on the same VM-Host. If one is migrated to an other host, the L2 cluster is broken.

But all maschines are always reachable by IP. And VRRP works also.

Only the L2 Connection is gone.

 

Must be something on the VM side, i think.

Any suggestions?

 

Thank you!

Hi,

 

Please double check if all VLAN's on the cluster are 'connected' all the way to the other esx host. If you don't excluded VLAN's they must all be constitent between the complete setup.

 

Hope this helps

 

Hello "hno", I have exactly the same issue, if both vMCs were hosted in the same VMware host the cluster works as expected, if the vMCs were hosted in separated VMware hosts, the cluster not works, the status is two vMCs in Isolated Leader. All other communication between vMCs are ok, ping, arp, vrrp, etc. Only the cluster not come up. Please if you have some manner to solve it post here. Since for full HA the two vMCs should not be running in the same VMware host.

Hello

Did you check:

1. Full L2 connectivity between the esx hosts
2. Forged transmits and promiscuous mode both on on all esx’s

Can you use a separate vswitch just for the vmc’s and vmm?

Yes I already checked full L2 communication, vrrp between vmc are ok too. Promiscuous, forged transmits and mac changes are enabled, Net.ReversePathPromisc also enabled too. I will try to use separate vswitch. Now I am using the same distributed vswitch for vmc. Tks.

What version of AOS on the VMC, and what version of ESX?

ESX 6.5

AOS 8.2.2.3 also tried 8.4.0.0 with same result.

I can not create a separate vswitch for each vMC because I do not have uplink for it, I tried to put each vMC in a separated vPortGroup but the result is the same.

 

Cluster Info Table
------------------

Type IPv4 Address Priority Connection-Type STATUS
---- --------------- -------- --------------- ------
self 172.26.1.101 128 N/A ISOLATED (Leader)
peer 172.26.1.102 128 N/A SECURE-TUNNEL-ESTABLISHED
(MC-01) #

 

 

Cluster Info Table
------------------
Type IPv4 Address Priority Connection-Type STATUS
---- --------------- -------- --------------- ------
peer 172.26.1.101 128 N/A SECURE-TUNNEL-ESTABLISHED
self 172.26.1.102 128 N/A ISOLATED (Leader)
(MC-02) #

 

Attached I put a capture from two vMCs, one thing I found that is not good is the fact of PAPI packets from one vMC are not arriving at the other one, another thing is that mac of PAPI packets have source/dest all 0s(00:00:00:00:00:00). The IP address information are correct, 802.1q information also are present in some ISAKMP packets, my management port is configured as access, in my mind this kind of packets should not have 802.1q info.

How are you checking full L2 connectivity between the two VMCs? 

 

I'm trying to carve out the time to stand up a second ESX box in my lab to recreate, but have not encountered this problem in production networks.

 

Can you add screen shots showing the VMC connectivity within the VM environment, as well as the interfaces as seen from AOS? The cluster profile would also be helpful, to verify any excluded VLANs listed there.

Bellow you will see L2 full communication between MCs. In your lab do you have 2 ESX separeted hosts?

My topology have HP virtual connect, I am think about some issue in it.

My Topology
ESXI1---vSwitch---VirtualConnect1---DellSwitchToR1---DellSwitchToR2---VirtualConnect2---vSwitch---ESXI2

Capture of packets made in the interfaces of Dell switches facing to VirtualConnect, I could see Aruba PAPI protocol from vMC1 and vMC2, in both directions, a capture made at vMC I saw only PAPI packets leaving but nothing arriving, I think VirtualConnect are discarding Aruba PAPI protocol for some reason, but ICMP packets and all other network protocol completes as expected. If both vMCs were hosted in the same ESXI(only for tests) the communication is made direct by vSwitch and all looks good, I am think that VirtualConnect have some issue to work with Aruba PAPI.

Any one here with knowledge of HP VirtualConnect?

What's the output of  

 

show lc-cluster vlan-probe status

 

Resolved my issue via the VLAN_FAIL column

You can also compare the configurations between MDs from your MM.

 

Here's an example in my lab.

(PDC-MM1-2.2) [mynode] #show configuration node-hierarchy

Default-node is not configured. Autopark is disabled.

Configuration node hierarchy
----------------------------
Config Node Type Name
----------- ---- ----
/ System
/md System
/md/testing123 Group
/md/The-Beatles Group
/md/The-Beatles/Cluster-1 Group
/md/The-Beatles/Cluster-1/00:0c:29:c1:fc:36 Device SDC-VMC2-Cluster1
/md/The-Beatles/Cluster-1/00:0c:29:dc:f4:ec Device PDC-VMC1-Cluster1
/md/The-Beatles/Cluster-2 Group
/md/The-Beatles/Cluster-2/00:0c:29:ec:5a:94 Device SDC-VMC2-Cluster2
/md/The-Beatles/Cluster-2/00:0c:29:fb:7a:ae Device PDC-VMC1-Cluster2
/mm System
/mm/mynode System

 

You can use the context attribute on the below command for additional clarity.
(PDC-MM1-2.2) [mynode] #show configuration diff /md/The-Beatles/Cluster-1/00:0c:29:dc:f4:ec /md/The-Beatles/Cluster-1/00:0c:29:c1:fc:36
interface vlan 60
- ip address 10.60.2.11 255.255.240.0
+ ip address 10.60.2.12 255.255.240.0
interface vlan 610
- no suppress-arp
- interface vlan 99
- ip address dhcp-client
vrrp 2
- preempt
- priority 200
- mgmt-user admin root 26e4596d013ae252c871a9f605e475906ab8c4fc5841980704 max-concurrent-sessions 0
+ mgmt-user admin root 0db05bef01e2e5c7b59adc41bbb2ee858e0a8dbba25463a087 max-concurrent-sessions 0
interface gigabitethernet 0/0/1
interface gigabitethernet 0/0/2
- switchport trunk allowed vlan 1,60
+ switchport trunk allowed vlan 1
- trusted vlan 1,60
+ trusted vlan 1
- hostname PDC-VMC1-Cluster1
+ hostname SDC-VMC2-Cluster1
(PDC-MM1-2.2) [mynode] #

 

The output of vlan-probe status looks good.

(MC-01) [MDC] #show lc-cluster vlan-probe status

Cluster VLAN Probe Status
-------------------------
Type IPv4 Address REQ-SENT REQ-FAIL ACK-SENT ACK-FAIL REQ-RCVD ACK-RCVD VLAN_FAIL CONN-TYPE START/STOP
---- --------------- -------- -------- -------- -------- -------- -------- --------- --------- ----------
peer 172.26.1.102 0 0 0 0 0 0 0 N/A 0/ 49
(MC-01) [MDC] #

 

(MC-02) [MDC] #show lc-cluster vlan-probe status

Cluster VLAN Probe Status
-------------------------
Type IPv4 Address REQ-SENT REQ-FAIL ACK-SENT ACK-FAIL REQ-RCVD ACK-RCVD VLAN_FAIL CONN-TYPE START/STOP
---- --------------- -------- -------- -------- -------- -------- -------- --------- --------- ----------
peer 172.26.1.101 0 0 0 0 0 0 0 N/A 0/ 49
(MC-02) [MDC] #

 

About config diffrent the only different configs that I have are ip address, priority of vrrp and hostname.

Ah I saw your previous screenshots. It appears its failing to establish the IPSec tunnel.

 

The traffic needed between MCs =

(IPSec Phase 1) ISAKMP UDP 500

(IPSec Phase 2) ESP IP Protocol 50

 

Truly make sure theres no fw or acls between.

 

MC-01:show crpyto isakmp sa

MC-01:show crypto ipsec sa

 

Setting up debug to see what's failing:

MM1 [md] (config) # change-config-node MC-01

config t

logging security process authmgr level debug

logging security process crypto level debug

logging system processs cluster_mgr level debugging

wr me

 

MC-01#  show log security 100

 

Buried in here will tell us if its Phase 1 or Phase 2. 

 

 

Hello Qlimax, as lc-cluster status suggest I can see that tunnel is stablished, in MC-01 I see tunnels with MM, AP and MC-02, in MC-02 I see tunnel with MM and MC-01. The output of logs you suggest to enable only show messages of rouge SSID detected, take a look:

 

Mar 25 11:08:24 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID a2:2c:36:b3:7e:23 and SSID MirrorCAST 36B37E23 on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
Mar 25 11:10:54 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID dc:bf:e9:3e:ab:0a and SSID Moto G (5S) 1017 on CHANNEL 4) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
Mar 25 11:11:58 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID 30:cb:f8:1c:79:87 and SSID AndroidAP on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
Mar 25 11:12:07 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID 38:9a:f6:84:30:69 and SSID AndroidAP3069 on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
Mar 25 11:12:45 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID b0:6e:bf:d7:4d:09 and SSID Euzebio on CHANNEL 11) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
Mar 25 11:13:19 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID 02:57:c1:ed:c7:89 and SSID tancredo on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.
Mar 25 11:13:31 :126005: <6273> <WARN> |wms| |ids| Interfering AP: The system classified an access point (BSSID fc:64:3a:b8:a2:58 and SSID AndroidAPA258 on CHANNEL 6) as interfering. Additional Info: Detector-AP-Name:TESTE_b0:b8:67:c8:56:ae; Detector-AP-MAC:b0:b8:67:05:6a:e0; Detector-AP-Radio:2.

Sorry, read through the history but didn't see note of it. You are seeing this when they are spread across two separate hypervisors but with the same dvSwitch correct? Fot the vSwitch in question, is it using two physical adapters on the vSwitch? If so, is LACP used between the hypervisors and the physical uplink switch or is it just using NIC Teaming?

jhoward, I am using just one NIC for vswitch, do not have lacp nor nic team. When using same hypervisor everything looks good, with no change of AOS config, I have in my mind that issue is something about compatibility with VMware enviroment, maybe.

Does your output still look the same in previous screenshots?

 

show lc-cluster group-membership

 

Are they stuck on isolated and "Secure tunnel negotiating"

 

If so the traffic between the MCs is either being filtered somehow or there is a misconfig in the IPSec config I'd think.

The role is stucked "Isolated (Leader)" and status keep moving between disconnected, negotiating and established. I do not have any filter in the path, just L2 network, vswitch, virtual connect switch and standard network switch, without any filter.

Is phase 1 or 2 breaking

Sent from my IPhone

Seems that phase 1 and 2 are stablished.

I dont believe so if they're stuck in the state of "Secure Tunnel Negotiating" 

show lc-cluster group-membership

 

Why it could be happening? Not sure...I have cluster members on different hosts. 

It stuck in "Secure-Tunnel-Stablished" stay some time in it, since not receives PAPI packets it go back to disconnect, then negotiating and again to stablished. It not stuck in negotiating, sorry if I am not clear enough.

FYI I had another issue with having my MCs not forming an L2 after adding a few vlans. Come to find out I missed some switchport tagging on redundant links to my hosts. Just another thing to check....

Just to update the case, we solve the issue creating 2 another new ESXi servers and migrate MC to these new servers, as a magic everthing working as expected now. We couldn't identify the rason for this issue, but looks like something specific with hardware HP Blade System with Virtual Connect.

Hi,

 

Please let us know the HPE Blade Server model number details.

 

HP BladeSystem C7000 witch Flex10 Module(VirtualConnect 4.63)

 

TAC concludes that around 99% of papi packets was lost

Many of the blade systems with their integrated virtual network stack behind it don't usually support promiscuous mode or forged transmits, and require usually a direct VM to physical port map of switch port to the virtual interface of the VMM or VMC. Either that or they don't handle a virtualized mac on the VM that doesn't directly map to a vMAC of the virtual interface on the VM. 

Hello Jhoward, the problem is that in this case we do not have available interface to dedicate for MDs, all physical NICs are in use.

Hi

Can you please update which HPE blade server was used ? CPU and memory and all details. 

and which VMC SKUs were being used in cluster?

Hi Airheads,

 

tomorrow we do a troubleshooting session with an Aruba WLAN- and an ESX Guy. 

I'll keep you informed.

 

best regards,

Michael