Wireless Access

Hi folks,

 

today I am trying to set up a LAB with a ESXi 6.5 but every time I start up the MM or VMC machines they crash at boot! This only happens with AOS 8.5 (tried 8.5.0.0 and 8.5.0.2) 8.3 and 8.4 are working.

System says "Datapath crash ... rebooting" right after "Starting OS Services"

Is this a know bug?

 

Thanks

Stefan

 

 

 

Try to install 8.5 clean , with out upgrade the VMM from 8.4 or 8.3 , And share with us if it worked.

( Does you VMM machine settings - cpu.mem.hdd - everyhing as Aruba recommended ? )

Regards.

Asa

Hi,

 

did a "clean" install with the .OVA (also tried to create a VM with the proper specs and installed using the .ISO.)

8.3 and 8.4 are working as expected. If I upgrade or install fresh it does not start and crashes into a boot loop

The Reason i advised it  (Clean install), Because i had the same issue , but when trying upgrading from VMM 8.4.0.X to 8.5.0.X , and TAC told me to try a clean upgrade - Because they didnt encouter the same error in their lab enviroment at all.

(Didnt had a chance to validate it - but your post like validiting it for me in a way)

 

Q:While deploying the VMM did u choose to encrypt all files?

 

If it may assist you:

(This is the case i opened with same datapath bootloop only in 8.5 VMM)

Case ID: 5338929101 - Aruba 7205 (IL) Controller-ArubaOS 8.5.0.0 MM crash on boot after upgrade.

 

 

Also,I know it might sound like a stupid question , But did you follow all the deployment guidelines from the following link:

 

ArubaOS 8.5.0.0 Virtual Appliance Installation Guide.pdf

https://support.arubanetworks.com/Documentation/tabid/77/DMXModule/512/Command/Core_Download/Default.aspx?EntryId=35435

 

 

 

 

 

 

hmmm

Sure I didn‘t do the reservation for RAM etc.... but what should be the difference between 8.4 and 8.5?

I will try updating ESX to U3 but this will be done in two weeks. I will report then

Thanks

 

Hi

 

I'm facing the same exact issue. MM is working like a dream on 8.4.0.4, but as soon as i try 8.5.x, it crashes on boot with the same errormessage.

I have tried importing a new .ova file to my ESXi, but this crashes on boot aswell.

I will try to upgrade my ESXI aswell, but i would like know if this helps before i move on to that.

 

Roar

Hi,

If you followed the above guide , and still got an error wtih 8.5.0.X VMM

 

Try upgrade your ESX , or contact TAC , Because you both reporting on an the same issue i encountered in one of my deployment - read above thread.

 

Asa.

did a upgrade to ESXi 6.5 U3 today. No luck. :-(

 

Contact TAC....:(  I have had 8.3, 8.4 and 8.5 running from NUCs to servers, but I followed all of the configuration requirements in the deployment guide and I have never had issues.  Even if you followed everything, you could have a corner case that TAC would be able to uncover.

Hi

 

I did contact TAC and got an explaination to as why it will not work on my small ESXi lab setup.

I have run alle versions from 8.0 up to 8.4.

As from 8.5 the system is checking NUMA nodes, this is what TAC wrote to me

 

The issue is seen when more than half the number of CPUs on ESXi are allocated to a VMM instance. For example, if the ESXI server has 12 CPUs and VMM instance is allocated >= 6 CPUs. This might be due to CPUs per socket configuration. On ESXi it was observed that when more than half the CPUs are allocated to a VMM instance then 2 NUMA nodes are assigned to the VMM. To override this set numa.autosize.vcpu.maxPerVirtualNode under VM advanced option to the number of CPUs allocated to the VMM. In this case only one NUMA node is assigned to the VMM and datapath  wont crash.

To answer the question of why datapath crash is seen when 2 NUMA nodes are assigned. With 8.5.0.0 dpdk, hugetables are equally divided to both NUMA socket. What I observed was, even though 2 NUMA nodes are assigned, the physical_id for all CPUs under /proc/cpuinfo is assigned to 0. When dpdk tries to allocate memory on socket 1 it fails. I guess  dpdk is not able to find the socket id 1. There seems to be a direct relation b/w socket to NUMA nodes.

 

This i guess is not an issue in larger ESXi installations, but my single server with sparse resources.

 

Roar 

Hi

I also contacted TAC. What you mentioned is not all! Also 8.5 requires INTEL SSE4 Support on the CPU side. This is the showstopper regarding my LAB. But anyways.... good to know!

What is your lab setup?

---

@snowman wrote:
Hi

I also contacted TAC. What you mentioned is not all! Also 8.5 requires INTEL SSE4 Support on the CPU side. This is the showstopper regarding my LAB. But anyways.... good to know!

---

This was very helpful in pointing us in the right direction.  We were experiencing this same crash/boot loop while trying to upgrade from 8.4.0.4 to 8.5.0.5.  As it turns out, our vcenter server was created many years ago on much older hardware and continually upgraded from there.  As a result our vcenter was still set to present "Penryn" generation CPUs to all virtual machines.  "Penryn" only supports SSE4.1 and the crash logs on the mobility master says it needs SSE4.2.  We had to change the setting in vcenter to present a newer generation CPU to virtual machines and then the mobility master booted right up (we're at "Skylake" now).  Apparently this is a setting that carries over through vcenter upgrades instead of changing based on the hardware.

 

For clarity, the setting in vcenter to check out is "VMware EVC".