Wireless Access

Reply
Highlighted
MVP Expert
MVP Expert

-Blog- ArubaOS AP Discovery through Firewall with VRRP

"I had an issue that was resolved together with TAC support, which I like to share with the community to share knowledge."

 

In a standard ArubaOS8 controller based solution the default AP to Controller communication can be placed both in the same layer 2 broadcast domain (management vlan) or can be routed, but in some cases it may be necessary to use a firewall between them.

 

AP is not comming UP – Basic Checks

  • Is the AP getting enough power?
  • Is the AP getting an IP address?
  • How is the AP connecting to the Aruba MC?
    • Verify if the AP is connection to the correct MC IP address
  • Does the AP have IP connectivity to the Aruba MC?
    • get console to AP and see if it can ping MC IP
  • Are the correct protocol ports opened between the AP and the MC?
    • Protocol ports UDP 69 (TFTP), UDP 123 (NTP),UDP 514 (SYSLOG), UDP 8211 (PAPI), TCP 20/21 (FTP), Protocol 47 (GRE) and UDP 500,4500 (IPSEC/IKE) need to be opened between AP en MC, in both directions.
  • Licensing limits or MC limits exceeded
  • User limits of Platform
  • Other configuration issues

Note: Control Plane Security (CPSEC) is enabled by default and secure the PAPI management traffic in an IPSEC tunnel, this is not documented in some older guides.

 

Note: It’s recommended that the firewall ports are opened in both directions. This is not documented in the Aruba User Guide, link (page 582). Most important when using VRRP for the AP discovery process especially for PAPI UDP 8211 traffic…

 

For better understanding of the AP boot process see this video; https://community.arubanetworks.com/t5/Video/AP-Boot-Sequence-amp-AP-Controller-Communication/ta-p/553567

 

Troubeshooting steps taken

After my AP’s get not successful pass the discovery process i start with basic troubleshooting following the above steps. This was easily done by using a serial console cable connected to the AP and following the boot process. The AP console shows my AP get an IP address and use DHCP option 43/60 to known the controller (VRRP) IP.

 

-	APBoot 1.0.9.12 (build 22797)
-	Built: 2009-11-04 at 15:53:54
-	
-	Model: AP-10x
-	CPU:   AR7161 revision: A2
-	Clock: 680 MHz, DDR clock: 340 MHz, Bus clock: 170 MHz
-	DRAM:  128 MB
-	POST1: passed
-	Copy:  done
-	Flash: 16 MB
-	PCI:   scanning bus 0 ...
-	       dev fn venID devID class  rev    MBAR0    MBAR1    MBAR2    MBAR3
-	       00  00  168c  0029 00002   01 10000000 00000000 00000000 00000000
-	       01  00  168c  0029 00002   01 10010000 00000000 00000000 00000000
-	Net:   eth0
-	Radio: ar922x#0, ar922x#1
-	
-	Hit <Enter> to stop autoboot:  0
-	Checking image @ 0xbf100000
-	Invalid image format version: 0xffffffff
-	eth0 up: 1 Gb/s full duplex
-	DHCP broadcast 1
-	*** Unhandled DHCP Option in OFFER/ACK: 2
-	*** Unhandled DHCP Option in OFFER/ACK: 42
-	*** Unhandled DHCP Option in OFFER/ACK: 224
-	*** Unhandled DHCP Option in OFFER/ACK: 2
-	*** Unhandled DHCP Option in OFFER/ACK: 42
-	*** Unhandled DHCP Option in OFFER/ACK: 224
-	DHCP IP address: 192.168.3.3
-	DHCP subnet mask: 255.255.255.0
-	DHCP def gateway: 192.168.3.254
-	DHCP DNS server: 172.16.200.1
-	DHCP DNS domain:
-	Controller address: 172.16.200.30
-	Using eth0 device
-	TFTP from server 172.16.200.30; our IP address is 192.168.3.3; sending through gateway 192.168.3.254
-	Filename 'mips32.ari'.
-	Load address: 0x2000000
-	Loading: #################################################################
-	         #################################
-	done
-	Bytes transferred = 6413400 (61dc58 hex)
-	
-	Image is signed; verifying checksum... passed
-	Signer Cert OK
-	Policy Cert OK
-	RSA signature verified.
-	Automatic boot of image at addr 0x02000000 ...
-	ELF file is 32 bit
-	Loading .text @ 0x80e00000 (6343640 bytes)
-	Loading .data @ 0x8140cbe0 (32 bytes)
-	Clearing .bss @ 0x8140cc00 (16 bytes)
-	## Starting application at 0x80e00000 ...
-	Uncompressing............................................................
-	
-	
-	Aruba Networks
-	ArubaOS Version 8.6.0.4 (build 74969 / label #74969)
-	Built by p4build@pr-hpn-build07 on 2020-04-02 at 14:20:28 UTC (gcc version 4.3.3)
-	CPU Rev: aa
-	71x CPU
-	Flash variant: default
-	Cache parity protection disabled
-	Using 340.000 MHz high precision timer. cycles_per_jiffy=680000
-	Memory: 119808k/131072k available (1742k kernel code, 11128k reserved, 989k data, 5504k init, 0k highmem)
-	available.
-	detected lzma initramfs
-	initramfs: LZMA lc=3,lp=0,pb=2,dictSize=8388608,origSize=27885056
-	LZMA initramfs by Ming-Ching Tiew <mctiew@yahoo.com> ..........................................................................................................................................................................................................................................................................................................................................................................................................................................
-	AR7100 GPIOC major 0
-	wdt: registered with refresh
-	Enabling Watchdog
-	Talisker RSSI LED initialization
-	Creating 1 MTD partitions on "ar7100-nor0":
-	0x00000000-0x01000000 : "flash"
-	i2c /dev entries driver
-	i2c-talisker: using default base 0x18040000
-	AD7416 driver probing for devices on AR7100 I2C
-	.<6>lo: Disabled Privacy Extensions
-	IPv6 over IPv4 tunneling driver
-	
-	Starting Kernel SHA1 KAT ...Completed Kernel SHA1 KAT
-	Starting Kernel HMAC-SHA1 KAT ...Completed Kernel HMAC-SHA1 KAT
-	Starting Kernel DES KAT ...Completed Kernel DES KAT
-	Starting Kernel AES KAT ...Completed Kernel AES KAT
-	
-	Starting Kernel AESGCM KAT ...Completed Kernel AESGCM KAT
-	
-	Populate AP type info
-	Domain Name: arubanetworks.com
-	CMDLINE_WRITE_ENVIRONMENT arg {num_reboot=3}
-	No panic info available
-	talisker: Start hotplug
-	Testing TPM... Passed
-	apfcutil: sector CLIENT: apfc_check failed: Not an APFC sector for this AP Type
-	apfcutil: apfc_init failed: Not an APFC sector for this AP Type
-	ag7100_mod: module license 'unspecified' taints kernel.
-	AG7100: Length per segment 512
-	AG7100: Max segments per packet 4
-	AG7100: Max tx descriptor count    400
-	AG7100: Max rx descriptor count    252
-	AG7100: fifo cfg 3 018001ff
-	AG7100CHH: Mac address for unit 0
-	AG7100CHH: 00:24:6c:cb:51:bc
-	AG7100: cfg1 0xf cfg2 0x7014
-	ATHRF1: Port 0, Neg Success
-	ATHRF1: unit 0 phy addr 0 ATHRF1: reg0 3100
-	ag7100_ring_alloc Allocated 4800 at 0x80874000
-	ag7100_ring_alloc Allocated 3024 at 0x872cf000
-	AG7100: cfg1 0xf cfg2 0x7014
-	ATHRF1: Port 0, Neg Success
-	ATHRF1: unit 0 phy addr 0 ATHRF1: reg0 3100
-	AG7100: unit 0: phy not up carrier 1
-	Writing 4
-	ADDRCONF(NETDEV_UP): bond0: link is not ready
-	/etc/init.d/rcS: 1075: cannot create /proc/sys/net/ipv6/conf/boinit_uol_mod: offload cap: 0x0, mesh mode none, strapless_enabled 0, uplink_vlan 0
-	nd0/accept_ra: Directory nonexistent
-	AP xml model 39, num_radios 2 (jiffies 15069)
-	apType 39 hw_opmode 0
-	radio 0: band 1 ant 0 max_ssid 8
-	radio 1: band 0 ant 0 max_ssid 8
-	init_asap_mod: installation:0
-	firewall cpu: core-0
-	AG7100: unit 0 phy is up...RGMii 1000Mbps full duplex
-	AG7100: pll reg 0x18050010: 0x110000  AG7100: cfg_1: 0x1ff0000
-	AG7100: cfg_2: 0x3ff
-	AG7100: cfg_3: 0x18001ff
-	AG7100: cfg_4: 0xffff
-	AG7100: cfg_5: 0xfffef
-	AG7100: done cfg2 0x7215 ifctl 0x0 miictrl 0x22
-	ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
-	set device anul0 mtu to 2000
-	Starting watchdog process...
-	LLDP not sent yet, DHCP is waiting
-	Starting DHCP
-	net.ipv4.conf.all.arp_notify = 1
-	Getting an IP address...
-	net.ipv4.conf.all.arp_notify = 0
-	net.ipv4.conf.br0.arp_notify = 1
-	192.168.3.3 255.255.255.0 192.168.3.254
-	Running ADP...Done. Master is 172.16.200.30
-	ath_hal: 0.9.17.1 (AR5416, AR9380, REGOPS_FUNC, PRIVATE_DIAG, WRITE_EEPROM, 11D)
-	ath_rate_atheros: Copyright (c) 2001-2005 Atheros Communications, Inc, All Rights Reserved
-	ath_rate_atheros: Aruba Networks Rate Control Algorithm
-	ath_dfs: Version 2.0.0
-	Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
-	ath_spectrum: Version 2.0.0
-	Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
-	ath_dev: Copyright (c) 2001-2007 Atheros Communications, Inc, All Rights Reserved
-	ath_pci: 0.9.4.5 (Atheros/multi-bss)
-	ath_attach: scn 804c0280 sc 80680000 ah 806c0000
-	wifi0: Base BSSID 00:24:6c:35:1b:c8, 8 available BSSID(s)
-	bond0 address=00:24:6c:cb:51:bc
-	br0 address=00:24:6c:cb:51:bc
-	wifi0: AP type AP-105, radio 0, max_bssids 8
-	wifi0: Atheros 9280: mem=0x10010000, irq=49 hw_base=0xb0010000
-	ath_attach: scn 85b40280 sc 85b60000 ah 85b80000
-	wifi1: Base BSSID 00:24:6c:35:1b:c0, 8 available BSSID(s)
-	bond0 address=00:24:6c:cb:51:bc
-	br0 address=00:24:6c:cb:51:bc
-	wifi1: AP type AP-105, radio 1, max_bssids 8
-	wifi1: Atheros 9280: mem=0x10000000, irq=48 hw_base=0xb0000000
-	ath_ahb: 0.9.4.5 (Atheros/multi-bss)
-	
-	Starting FIPS KAT ... Completed FIPS KAT
-	
-	keep watchdog process alive for talisker (nanny will restart it)...
-	
-	        <<<<<       Welcome to the Access Point     >>>>>
-	
-	~ #

From the AP console I tested whether the MC’s and VRRP could be reached using ping. No issues so far…

 

~ # ping 172.16.200.90 (VRRP)
PING 172.16.200.90 (172.16.200.90): 56 data bytes
64 bytes from 172.16.200.90: icmp_seq=0 ttl=64 time=9.0 ms
64 bytes from 172.16.200.90: icmp_seq=1 ttl=64 time=3.1 ms

~ # ping 172.16.200.91 (MC01)
PING 172.16.200.91 (172.16.200.91): 56 data bytes
64 bytes from 172.16.200.91: icmp_seq=0 ttl=64 time=1.4 ms
64 bytes from 172.16.200.91: icmp_seq=1 ttl=64 time=2.1 ms

~ # ping 172.16.200.92 (MC02)
PING 172.16.200.92 (172.16.200.92): 56 data bytes
64 bytes from 172.16.200.92: icmp_seq=0 ttl=64 time=2.1 ms
64 bytes from 172.16.200.92: icmp_seq=1 ttl=64 time=2.4 ms

Next test is to see if the PAPI (UDP 8211) management packets can reach the controller with the master VRRP. For this test i login on the MC controller through the Mobility Master. And yes we see some PAPI messages arriving.

 

(HomeLAB-TST01) [MDC] #show datapath session table | include 192.168.3.3
172.16.200.91     192.168.3.3     17   8222  8211   0/0     0    0   1   local       8    2          400        FCI             2
192.168.3.3       172.16.200.91   17   8211  8222   0/0     0    0   0   local       8    0          0          FYI             2

And still my AP will not success the discovery process. In my CPSEC configuration i have “auto cert provisioning” enabled. And there are enough licenses in place, that is out of scope of this blog. So i start to look at the logfiles and found that the AP is unable to contact the switch (controller) and missing the HELLO messages from the controller.

 

(HomeLAB-TST01) [MDC] #show log all | include 00:24:6c:cb:51:bc
May 22 00:46:54  nanny[1185]: <303022> <WARN> |AP 00:24:6c:cb:51:bc@192.168.3.3 nanny|  Reboot Reason: AP rebooted Fri Dec 31 16:44:39 PST 1999; SAPD: Unable to contact switch: HELLO-TIMEOUT. Last rebootstrap reason: HELLO-TIMEOUT, 229 sec before: Last Ctrl msg: HELLO len=421 dest=172.16.200.90 tries=10 seq=0

Solution

When the AP use the VRRP address to contact the controllers the PAPI traffic contains <src:192.168.3.3 dest.172.16.200.90> when reaching the VRRP virtual IP, VRRP will forward the traffic to the controller that has the VRRP MASTER IP 172.16.200.91. The controller will answer the PAPI traffic with his own REAL IP address <src.172.16.200.91 dest.192.168.3.3. Because the source address is changed, this traffic must be considered as a new session.

Most firewalls are state-full, what means that return traffic from an initiated session is allowed in the same session, but therefore the source en destination IP address must be the same. Because the controller talk back with is own REAL IP as source IP, this cannot be handled by the same session.

 

We can see this process with a packet sniffer on the firewall.

firewall.jpg

I talked about this case with Aruba TAC Support and they recommend to open the required ports in both direction. However in my case it was enough to only add 8211 UDP in both directions.

 

firewall.JPG

 

 

Kind Regards Marcel Koedijk
HPE ASE Flexnetwork | ACMP | ACCP | Ekahau ECSE Design - Was this post usefull, Kudos are welcome.
Guru Elite

Re: -Blog- ArubaOS AP Discovery through Firewall with VRRP

Awesome writeup!

 

The default recommendation is to deploy APs whose ip addresses are routable to the management ip address to a controller, not on the same layer 2 subnet as the controller.

The firewall ports specification does not specify what are the source or destination ports, so the implication is "both"  https://www.arubanetworks.com/techdocs/ArubaOS_86_Web_Help/Content/arubaos-solutions/external-firewallconf/fire-port-conf-arub.htm#

 

I would reconfigure my network to allow APs direct connectivity to the ip addresses of controllers, rather than to get a firewall admin involved...  Rulesets change and any of those can break connectivity.


*Answers and views expressed by me on this forum are my own and not necessarily the position of Aruba Networks or Hewlett Packard Enterprise.*
ArubaOS 8.5 User Guide
InstantOS 8.5 User Guide
Airheads Knowledgebase
Airheads Learning Videos
Remote Access Point Solution Guide
ArubaOS Consolidated Release Notes
ArubaOS 8 ViA VPN Solution Guide
Highlighted
MVP Expert
MVP Expert

Re: -Blog- ArubaOS AP Discovery through Firewall with VRRP

Thanks Collin!

 

The customer have a shared datacenter with one other customer, and i'am not allowed to route on the coreswitch. On the other end they have 15 very small sites where i dont like to use a stretched vlan between sites. The single point for routing to keep traffic separate is the HA firewall. All together it isn't a heavy used environment, around 100-200 concurrent users. But sure, we have to respect the max. firewall throughput.

 

Kind Regards Marcel Koedijk
HPE ASE Flexnetwork | ACMP | ACCP | Ekahau ECSE Design - Was this post usefull, Kudos are welcome.
Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: