"I had an issue that was resolved together with TAC support, which I like to share with the community to share knowledge."
In a standard ArubaOS8 controller based solution the default AP to Controller communication can be placed both in the same layer 2 broadcast domain (management vlan) or can be routed, but in some cases it may be necessary to use a firewall between them.
AP is not comming UP – Basic Checks
- Is the AP getting enough power?
- Is the AP getting an IP address?
- How is the AP connecting to the Aruba MC?
- Verify if the AP is connection to the correct MC IP address
- Does the AP have IP connectivity to the Aruba MC?
- get console to AP and see if it can ping MC IP
- Are the correct protocol ports opened between the AP and the MC?
- Protocol ports UDP 69 (TFTP), UDP 123 (NTP),UDP 514 (SYSLOG), UDP 8211 (PAPI), TCP 20/21 (FTP), Protocol 47 (GRE) and UDP 500,4500 (IPSEC/IKE) need to be opened between AP en MC, in both directions.
- Licensing limits or MC limits exceeded
- User limits of Platform
- Other configuration issues
Note: Control Plane Security (CPSEC) is enabled by default and secure the PAPI management traffic in an IPSEC tunnel, this is not documented in some older guides.
Note: It’s recommended that the firewall ports are opened in both directions. This is not documented in the Aruba User Guide, link (page 582). Most important when using VRRP for the AP discovery process especially for PAPI UDP 8211 traffic…
For better understanding of the AP boot process see this video; https://community.arubanetworks.com/t5/Video/AP-Boot-Sequence-amp-AP-Controller-Communication/ta-p/553567
Troubeshooting steps taken
After my AP’s get not successful pass the discovery process i start with basic troubleshooting following the above steps. This was easily done by using a serial console cable connected to the AP and following the boot process. The AP console shows my AP get an IP address and use DHCP option 43/60 to known the controller (VRRP) IP.
- APBoot 1.0.9.12 (build 22797)
- Built: 2009-11-04 at 15:53:54
-
- Model: AP-10x
- CPU: AR7161 revision: A2
- Clock: 680 MHz, DDR clock: 340 MHz, Bus clock: 170 MHz
- DRAM: 128 MB
- POST1: passed
- Copy: done
- Flash: 16 MB
- PCI: scanning bus 0 ...
- dev fn venID devID class rev MBAR0 MBAR1 MBAR2 MBAR3
- 00 00 168c 0029 00002 01 10000000 00000000 00000000 00000000
- 01 00 168c 0029 00002 01 10010000 00000000 00000000 00000000
- Net: eth0
- Radio: ar922x#0, ar922x#1
-
- Hit <Enter> to stop autoboot: 0
- Checking image @ 0xbf100000
- Invalid image format version: 0xffffffff
- eth0 up: 1 Gb/s full duplex
- DHCP broadcast 1
- *** Unhandled DHCP Option in OFFER/ACK: 2
- *** Unhandled DHCP Option in OFFER/ACK: 42
- *** Unhandled DHCP Option in OFFER/ACK: 224
- *** Unhandled DHCP Option in OFFER/ACK: 2
- *** Unhandled DHCP Option in OFFER/ACK: 42
- *** Unhandled DHCP Option in OFFER/ACK: 224
- DHCP IP address: 192.168.3.3
- DHCP subnet mask: 255.255.255.0
- DHCP def gateway: 192.168.3.254
- DHCP DNS server: 172.16.200.1
- DHCP DNS domain:
- Controller address: 172.16.200.30
- Using eth0 device
- TFTP from server 172.16.200.30; our IP address is 192.168.3.3; sending through gateway 192.168.3.254
- Filename 'mips32.ari'.
- Load address: 0x2000000
- Loading: #################################################################
- #################################
- done
- Bytes transferred = 6413400 (61dc58 hex)
-
- Image is signed; verifying checksum... passed
- Signer Cert OK
- Policy Cert OK
- RSA signature verified.
- Automatic boot of image at addr 0x02000000 ...
- ELF file is 32 bit
- Loading .text @ 0x80e00000 (6343640 bytes)
- Loading .data @ 0x8140cbe0 (32 bytes)
- Clearing .bss @ 0x8140cc00 (16 bytes)
- ## Starting application at 0x80e00000 ...
- Uncompressing............................................................
-
-
- Aruba Networks
- ArubaOS Version 8.6.0.4 (build 74969 / label #74969)
- Built by p4build@pr-hpn-build07 on 2020-04-02 at 14:20:28 UTC (gcc version 4.3.3)
- CPU Rev: aa
- 71x CPU
- Flash variant: default
- Cache parity protection disabled
- Using 340.000 MHz high precision timer. cycles_per_jiffy=680000
- Memory: 119808k/131072k available (1742k kernel code, 11128k reserved, 989k data, 5504k init, 0k highmem)
- available.
- detected lzma initramfs
- initramfs: LZMA lc=3,lp=0,pb=2,dictSize=8388608,origSize=27885056
- LZMA initramfs by Ming-Ching Tiew <mctiew@yahoo.com> ..........................................................................................................................................................................................................................................................................................................................................................................................................................................
- AR7100 GPIOC major 0
- wdt: registered with refresh
- Enabling Watchdog
- Talisker RSSI LED initialization
- Creating 1 MTD partitions on "ar7100-nor0":
- 0x00000000-0x01000000 : "flash"
- i2c /dev entries driver
- i2c-talisker: using default base 0x18040000
- AD7416 driver probing for devices on AR7100 I2C
- .<6>lo: Disabled Privacy Extensions
- IPv6 over IPv4 tunneling driver
-
- Starting Kernel SHA1 KAT ...Completed Kernel SHA1 KAT
- Starting Kernel HMAC-SHA1 KAT ...Completed Kernel HMAC-SHA1 KAT
- Starting Kernel DES KAT ...Completed Kernel DES KAT
- Starting Kernel AES KAT ...Completed Kernel AES KAT
-
- Starting Kernel AESGCM KAT ...Completed Kernel AESGCM KAT
-
- Populate AP type info
- Domain Name: arubanetworks.com
- CMDLINE_WRITE_ENVIRONMENT arg {num_reboot=3}
- No panic info available
- talisker: Start hotplug
- Testing TPM... Passed
- apfcutil: sector CLIENT: apfc_check failed: Not an APFC sector for this AP Type
- apfcutil: apfc_init failed: Not an APFC sector for this AP Type
- ag7100_mod: module license 'unspecified' taints kernel.
- AG7100: Length per segment 512
- AG7100: Max segments per packet 4
- AG7100: Max tx descriptor count 400
- AG7100: Max rx descriptor count 252
- AG7100: fifo cfg 3 018001ff
- AG7100CHH: Mac address for unit 0
- AG7100CHH: 00:24:6c:cb:51:bc
- AG7100: cfg1 0xf cfg2 0x7014
- ATHRF1: Port 0, Neg Success
- ATHRF1: unit 0 phy addr 0 ATHRF1: reg0 3100
- ag7100_ring_alloc Allocated 4800 at 0x80874000
- ag7100_ring_alloc Allocated 3024 at 0x872cf000
- AG7100: cfg1 0xf cfg2 0x7014
- ATHRF1: Port 0, Neg Success
- ATHRF1: unit 0 phy addr 0 ATHRF1: reg0 3100
- AG7100: unit 0: phy not up carrier 1
- Writing 4
- ADDRCONF(NETDEV_UP): bond0: link is not ready
- /etc/init.d/rcS: 1075: cannot create /proc/sys/net/ipv6/conf/boinit_uol_mod: offload cap: 0x0, mesh mode none, strapless_enabled 0, uplink_vlan 0
- nd0/accept_ra: Directory nonexistent
- AP xml model 39, num_radios 2 (jiffies 15069)
- apType 39 hw_opmode 0
- radio 0: band 1 ant 0 max_ssid 8
- radio 1: band 0 ant 0 max_ssid 8
- init_asap_mod: installation:0
- firewall cpu: core-0
- AG7100: unit 0 phy is up...RGMii 1000Mbps full duplex
- AG7100: pll reg 0x18050010: 0x110000 AG7100: cfg_1: 0x1ff0000
- AG7100: cfg_2: 0x3ff
- AG7100: cfg_3: 0x18001ff
- AG7100: cfg_4: 0xffff
- AG7100: cfg_5: 0xfffef
- AG7100: done cfg2 0x7215 ifctl 0x0 miictrl 0x22
- ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
- set device anul0 mtu to 2000
- Starting watchdog process...
- LLDP not sent yet, DHCP is waiting
- Starting DHCP
- net.ipv4.conf.all.arp_notify = 1
- Getting an IP address...
- net.ipv4.conf.all.arp_notify = 0
- net.ipv4.conf.br0.arp_notify = 1
- 192.168.3.3 255.255.255.0 192.168.3.254
- Running ADP...Done. Master is 172.16.200.30
- ath_hal: 0.9.17.1 (AR5416, AR9380, REGOPS_FUNC, PRIVATE_DIAG, WRITE_EEPROM, 11D)
- ath_rate_atheros: Copyright (c) 2001-2005 Atheros Communications, Inc, All Rights Reserved
- ath_rate_atheros: Aruba Networks Rate Control Algorithm
- ath_dfs: Version 2.0.0
- Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
- ath_spectrum: Version 2.0.0
- Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
- ath_dev: Copyright (c) 2001-2007 Atheros Communications, Inc, All Rights Reserved
- ath_pci: 0.9.4.5 (Atheros/multi-bss)
- ath_attach: scn 804c0280 sc 80680000 ah 806c0000
- wifi0: Base BSSID 00:24:6c:35:1b:c8, 8 available BSSID(s)
- bond0 address=00:24:6c:cb:51:bc
- br0 address=00:24:6c:cb:51:bc
- wifi0: AP type AP-105, radio 0, max_bssids 8
- wifi0: Atheros 9280: mem=0x10010000, irq=49 hw_base=0xb0010000
- ath_attach: scn 85b40280 sc 85b60000 ah 85b80000
- wifi1: Base BSSID 00:24:6c:35:1b:c0, 8 available BSSID(s)
- bond0 address=00:24:6c:cb:51:bc
- br0 address=00:24:6c:cb:51:bc
- wifi1: AP type AP-105, radio 1, max_bssids 8
- wifi1: Atheros 9280: mem=0x10000000, irq=48 hw_base=0xb0000000
- ath_ahb: 0.9.4.5 (Atheros/multi-bss)
-
- Starting FIPS KAT ... Completed FIPS KAT
-
- keep watchdog process alive for talisker (nanny will restart it)...
-
- <<<<< Welcome to the Access Point >>>>>
-
- ~ #From the AP console I tested whether the MC’s and VRRP could be reached using ping. No issues so far…
~ # ping 172.16.200.90 (VRRP)
PING 172.16.200.90 (172.16.200.90): 56 data bytes
64 bytes from 172.16.200.90: icmp_seq=0 ttl=64 time=9.0 ms
64 bytes from 172.16.200.90: icmp_seq=1 ttl=64 time=3.1 ms
~ # ping 172.16.200.91 (MC01)
PING 172.16.200.91 (172.16.200.91): 56 data bytes
64 bytes from 172.16.200.91: icmp_seq=0 ttl=64 time=1.4 ms
64 bytes from 172.16.200.91: icmp_seq=1 ttl=64 time=2.1 ms
~ # ping 172.16.200.92 (MC02)
PING 172.16.200.92 (172.16.200.92): 56 data bytes
64 bytes from 172.16.200.92: icmp_seq=0 ttl=64 time=2.1 ms
64 bytes from 172.16.200.92: icmp_seq=1 ttl=64 time=2.4 ms
Next test is to see if the PAPI (UDP 8211) management packets can reach the controller with the master VRRP. For this test i login on the MC controller through the Mobility Master. And yes we see some PAPI messages arriving.
(HomeLAB-TST01) [MDC] #show datapath session table | include 192.168.3.3
172.16.200.91 192.168.3.3 17 8222 8211 0/0 0 0 1 local 8 2 400 FCI 2
192.168.3.3 172.16.200.91 17 8211 8222 0/0 0 0 0 local 8 0 0 FYI 2
And still my AP will not success the discovery process. In my CPSEC configuration i have “auto cert provisioning” enabled. And there are enough licenses in place, that is out of scope of this blog. So i start to look at the logfiles and found that the AP is unable to contact the switch (controller) and missing the HELLO messages from the controller.
(HomeLAB-TST01) [MDC] #show log all | include 00:24:6c:cb:51:bc
May 22 00:46:54 nanny[1185]: <303022> <WARN> |AP 00:24:6c:cb:51:bc@192.168.3.3 nanny| Reboot Reason: AP rebooted Fri Dec 31 16:44:39 PST 1999; SAPD: Unable to contact switch: HELLO-TIMEOUT. Last rebootstrap reason: HELLO-TIMEOUT, 229 sec before: Last Ctrl msg: HELLO len=421 dest=172.16.200.90 tries=10 seq=0
Solution
When the AP use the VRRP address to contact the controllers the PAPI traffic contains <src:192.168.3.3 dest.172.16.200.90> when reaching the VRRP virtual IP, VRRP will forward the traffic to the controller that has the VRRP MASTER IP 172.16.200.91. The controller will answer the PAPI traffic with his own REAL IP address <src.172.16.200.91 dest.192.168.3.3. Because the source address is changed, this traffic must be considered as a new session.
Most firewalls are state-full, what means that return traffic from an initiated session is allowed in the same session, but therefore the source en destination IP address must be the same. Because the controller talk back with is own REAL IP as source IP, this cannot be handled by the same session.
We can see this process with a packet sniffer on the firewall.
I talked about this case with Aruba TAC Support and they recommend to open the required ports in both direction. However in my case it was enough to only add 8211 UDP in both directions.
