Thanks - we were not returning option 60 with option 43 in the DHCP reply. Now we are these have come back to life. Perhaps interestingly even if option 43 returns the wrong address it's enough for the DNS resolution of aruba-master to set the master properly. It appears to be a known issue with 8.7 upwards.
The documentation is quite interesting as in some places it seems as if the DNS resolution solution is preferred however elsewhere it's via option 43 & 60. I think we will move to both as a belt and braces solution.
------------------------------
Richard du Feu
------------------------------
Original Message:
Sent: Jan 20, 2021 10:08 AM
From: Cody Ensanian
Subject: AP-515, AOS8.7.1.1 in a reboot loop
Darn... well it sounds like you troubleshot and went down same paths we did... Bummer option 43 didn't help, that's odd. Only other things I could remotely think of is...
- did you also add option 60 to dhcp?
- did you pcap to make sure the options are being returned to the AP?
Else you might have to open a tac case.
------------------------------
Cody Ensanian
Original Message:
Sent: Jan 20, 2021 10:02 AM
From: Richard du Feu
Subject: AP-515, AOS8.7.1.1 in a reboot loop
Thanks. That sounds very similar. Interestingly we've tried option 43 and it's not helped and the other thing we've done is changed the DNS so that aruba-master resolves to all our controllers rather than the VIP. Neither of these have fixed our problem.
We've packet captured the dns traffic and it is returning aruba-master properly.
------------------------------
Richard du Feu
Original Message:
Sent: Jan 20, 2021 09:54 AM
From: Cody Ensanian
Subject: AP-515, AOS8.7.1.1 in a reboot loop
We had a similar issue when upgrading to 8.7.0.0.... aruba-master seemed to resolve fine on the network, but the APs just wouldn't set their master properly (we saw what you are seeing, the "Running ADP...Done. Master is"). Not saying you are having the same exact issue, but its very similar (for us, it effected ALL of our APs)
You could also packet capture between the AP and the uplink switch to confirm for sure your dns server is returning aruba-master properly.
Our APs were getting the aruba-master resolution back just fine, but the APs just wouldn't set the master (bug?). Our "quick fix" was to enable option 43 on the dhcp scope, and that fed the APs the proper IP and that worked to get us online.
We're still trying to figure out "why" or is it a bug in 8.7.0.0. TAC wasnt too helpful. We recently upgraded to 8.7.1.1 but havnt had time to remove option 43 from a test scope to see if ADP works properly now by resolving aruba-master.
------------------------------
Cody Ensanian
Original Message:
Sent: Jan 19, 2021 07:15 AM
From: Richard du Feu
Subject: AP-515, AOS8.7.1.1 in a reboot loop
Good morning,
I'm pretty new to Aruba WiFi as we're in the middle of a migration from Cisco so sorry if I get some terminology wrong here!
We had to upgrade our 7210 controller estate to 8.7 to enable support for AP-503H APs, perhaps mistakenly I went for the latest and greatest (8.7.1.1). The process worked pretty well with preuploading the images and the sequential upgrades limiting outage except for 10 of our ~60 AP-515s did not rejoin the controllers post firmware. We rolled back to 8.6.0.6 to see if it was a specific issue to 8.7.1.1. I've pulled one of the affected AP-515s out, done a factory default reset and it just worked fine rejoining the controllers, pulling it's config and downgrading to 8.6.0.6 without a hitch, additionally it had upgraded to 8.7.1.1 however it was not joining the controller properly. The serial output shows nothing too exciting except things around the master selection:
net.ipv4.conf.all.arp_notify = 1
Getting an IP address...
[ 26.011530] device eth0 entered promiscuous mode
[ 30.504330] Enabling USB power
[ 30.571121] POE power conditions have improved: the new condition is 'POE-AT: No restrictions'
net.ipv4.conf.all.arp_notify = 0
net.ipv4.conf.br0.arp_notify = 1
10.1.1.67 255.255.240.0 10.1.0.1
Running ADP...Done. Master is
Starting Webserver
bind: Address already in use
[ 44.784571] usbcore: registered new interface driver usbserial
[ 44.847324] usbcore: registered new interface driver usbserial_generic
[ 44.925547] usbserial: USB Serial support registered for generic
[ 45.004076] usbcore: registered new interface driver cp210x
[ 45.064299] usbserial: USB Serial support registered for cp210x
[ 45.177086] usbcore: registered new interface driver cdc_eem
master is changed from 0 to , cleanup cached info for old master
AP rebooted Wed Dec 31 16:01:44 PST 1969; Could not get the master ip address
shutting down watchdog process (nanny will restart it)...
<<<<< Welcome to the Access Point >>>>>
password: The system is going down NOW !!
Sending SIGTERM to all processes.
[ 113.010977] PMM: Failed to send netlink event:-3
Sending SIGKILL[ 115.694951] UBIFS (ubi1:4): un-mount UBI device 1
[ 115.744066] UBIFS (ubi1:4): background thread "ubifs_bgt1_4" stopsUnmounting UBIFS completed.
Please stand by while rebooting the system.
[ 117.905571] reboot: Restarting system
Comparing this to the output from a config reset AP it seems to be around master selection:
Running ADP...Done. Master is 10.1.32.11
Our DNS entry for aruba-master is correctly set and being returned to the APs.
The 10 AP-515s just sit there in an endless reboot loop to which my only solution at the moment is to go round them all and do a factory reset on them on the console port. Normally this wouldn't be a problem however access to some of these is heavily restricted thanks to Covid restrictions.
Anyway, a few of questions:
1 - Any idea what I could have done wrong in the firmware process to cause these APs to get so utterly confused a factory reset is the solution?
2 - Is there a way to reset these without a visit, particularly as they have no ssh enabled.
3 - Is this likely to be a 8.7.1.1 issue or something that occasionally happens with Aruba firmware upgrades?
Many thanks in advance.
------------------------------
Richard du Feu
------------------------------