Wireless Access

Hi,

I am having problem with VMC on vmware. Clients can't get IP address from DHCP but if I give them static IP addesses it works.

Long story: I converted one of our IAP-135 to CAP and deployed WMC on vmware to evaluate new ArubaOS 8. I created a test WLAN with 802.1x authentication. I use builtin "authenticated" role and vlan 20 for this test WLAN. Everything seems to work except dhcp. I use our corporate dhcp server. I use IP helper on the router for vlan 20.

I did packet capture on dhcp server. I can see that DHCP DISCOVER packets reach to dhcp server from routers relay ip and dhcp server sents DHCP OFFER packets back to router. I also hooked a VM to same port group on vswitch that VMC uses and do packet capture on it. So I see DHCP OFFER packets reach back to VMC. But If I do packet capture on the client I don't see any OFFER packet. I also did a packet capture on controller with "packet-capture datapath wifi-client <mac adress> " and there is no OFFER packets on this capture too.

So It seems that the dhcp server reply packets reach to VMC but not reach to clients. What should be the problem?

Thanks

Rahman

the vSwitch the VMC is on has promiscuous mode enabled? Might try to enabled forged transmits as well (would usually only be needed if VRRP were in use, but try anyway). 

Sure, promiscuous mode is enabled with the other two options. How can I debug it further? It is a pity that I can not do a packet capture on the physical interface (GigabitEthernet 0/0/1) of the VMC. VMC also says it does not support port mirroring (monitoring). So I see the vSwitch port group receives the dhcp server reply but I can not see what does the controller do with that packet.

Thanks,

Rahman

You can try to do an internal packet capture, and then pull the flashbackup and look at the pcap. Check the CLI and user guide

(HH-VMC1) #packet-capture

controlpath             Enable controlpath capture. Captured packets are stored

                        in /var/log/oslog/filter.pcap. Only capture to

                        local-filesystem is supported.

copy-to-flash           Copy captured packets to flash.

datapath                Enable datapath capture. Captured packets are stored in

                        /var/log/oslog/datapath.pcap or mirrored out of the

                        controller.

destination             Configure capture destination.

reset-pcap              Delete old pcap files and restart active capture.

#packet-capture controlpath udp all

#packet-capture copy-to-flash

Then on web interface I created flashbackup.tar.gz and copied it to my pc with scp. But I can't find any pcap files in it.

I will try to replicate and see if I get any pecap files in (havne't tried with 8.x). You can open a TAC case or reach out to your local Aruba SE if you aren't getting headway made. I've done dozens of VMC deployments, and no network issues, so there's likely something in the datapath, just hard to know where without getting hands on.

Well, local SE does not provide any support. They say ArubaOS 8 is still not GA and we should wait or consider buying HW Controllers.

I am also not very satisfied with the HPe TAC support. I opened two cases just to obtain Eval keys for VMC. One of the TAC technician insisted that I need to deploy virtual MM as VMC can't run in standalone mode and can't terminate APs on VMC wthout MM. Which I learned by my self that this is wrong and VMC can terminate APs in standalone mode.

I opened a new HPe TAC support case today. Hope I get the help I need this time.

Thanks for your helps.

Rahman

Hi ,

Today I had a remote session with HPe TAC. We find the problem. If the vSwitch of Vmware has more than one physical nic for redundancy (be it active-active or active-standby) VMC trunking mode or access mode somehow breaks DHCP for clients. If ıt has only one physical nic everything works as expected.

The support engineer captured control-path traffic and get all the logs for inspection. Can you test it to see if you can reproduce it? It seems an ArubaOs bug to me but we will see.

Thanks,

Rahman

I don't currently have a dual nic ESX setup, but if they find an issue or unexpected behavior, they should open a Bug and will fiel with engineering. If you want you can PM me your TAC Case ID. 

We are having the same problem with the VMC 8.2 version, we have the aggregation on the VMware(2 NICs on one Vswitch) and using trunk on one of the the VMC port.

And the internal DHCP server is not working too, I suspect that is related to the same problem. I cannot use the tunnel mode or Remote APs, I will try a workaround by put 1 vlan per port.

Anyone was able to overcome this problem ?

Hi

Have you been able to resolve this problem ?

I work on this some time and it seems that realy there is something with more than 1 NIC on vswitch 

regards

Karol

Hello.

We had the same problem and we solved this problem with this procedure:

On VMware ESXi you need an additional setting when using promiscuous mode, if vSwitch has more than one phys. NIC connected: "Net.ReversePathFwdCheckPromisc" must be set to 1. You will find it on Configuration - Software - Advanced Settings.

Hypervisor users (Especially VMware ESX/ESXi)

The below settings are specifically for VMware ESX/ESXi but similar settings may be present on Hyper-V, VirtualBox, and other similar hypervisors.

1. Enable promiscuous mode on the vSwitch
2. Enable MAC Address changes
3. Enable Forged transmits
4. If multiple physical ports exist on the same vswitch, the Net.ReversePathFwdCheckPromisc option must be enabled to work around a vswitch bug where multicast traffic will loop back to the host

Reference:

https://www.netgate.com/docs/pfsense/highavailability/troubleshooting-high-availability-clusters.html

https://forum.opnsense.org/index.php?topic=7206.0

Hi 

Thanks for suggestion 

We haven't tried this parameter.

In meantime we have reconfigure ports on phisical switch connecting ESX server. 

There wasn't any aggregation configured on the switch, and when we configured HP trunk on ports it started to work 

I had to test also the parameter you suggest by the way 

Best regards

Karol

There are cases where in VMC has only one active uplink (eg: 0/0/0), but the vswitch/dvswitch to which 0/0/0 is connected has multiple uplinks/NIC teaming (ESXI host to physical switch). The multiple uplinks need to be part of a LACP to prevent the uplink switch from looping back broadcast packets from one link to the redundant link (with both links being part of the NIC teaming). If LACP cannot be enabled, ReversePathFwdCheckPromisc setting in the ESXI can be used to prevent the vswitch/dvswitch from sending looped back packets to VMC.

Both LACP and ReversePathFwdCheckPromisc settings have been documented in VMC installation guide.

Reference:

https://www.arubanetworks.com/techdocs/ArubaOS_83_Web_Help/Content/PDFs/ArubaOS%208.3.0.0_Virtual%20Appliance%20Installation%20Guide.pdf

Update to this issue: We have also seen issues with few NIC cards looping back packets even if a single VMC port is used and there is no NIC teaming. Please always use the latest NIC drivers and firmwares. Please check the release notes of the NIC drivers for known issues and work around.

HI

Thanks very much and sorry for late answer

I know about LACP/aggregation on ports at switch side but I haven't seen this ReversePathFwdCheckPromisc parameter in VMware 

I see that it was added do documentation in 8.3.x.x version 

I will test it on my environment 

regards

Karol