Controllerless Networks

There is an issue I've come across that I've opened a case for, but wanted to start some technical discussion on. Bare with me as I want to be detailed in describing my situation.

• At times the IAP virtual controller IP address will become unresponsive for a cluster.
• Trying to access the GUI will fail, and pings to the VC IP address will fail.
• Pings to the subnet default gateway are successful
• No IP conflict issue & VC IP address is removed from DHCP scope
• Pings from gateway to VC IP address fail (of course, no ARP entry at switch with default gateway)
• Pings from an AP within the cluster succeed
  • result in ARP entry at switch with default gateway
  • pings from default gateway to VC IP address are now successful
  • pings from external network still fail
• Clear arp table on switch and go back to pings from gateway to VC IP address fail
• Reboot of master AP did not resolve the issue
• Reboot of AP cluster resolved the issue

This ultimately seems like a ARP and/or routing issue (reaching external networks via the default gateway). The current design does not mirror Aruba recommendations (AP management VLAN) and we are making changes to do so. However, how can this guarantee that these same issues don't appear when we make these changes? I've verified all configuration, and it is correct (Uplink vlan, switch (ProCurve) port configuration, VC IP configuration).

Sorry, I don't have packet captures as this was a remote site from where I am at.

Could you give us some more information about your cluster?  What type of IAPs, code version running, size of cluster etc?

I've seen issues with VC response in clusters that are very large (ie. over  ~80 members) and have a lot of multicast traffic riding the network.

AP-335

6.5.0.0-4.3.0.0_56428

14 APs in cluster.

Hi,

I have the same issue, on 6.4.4.8 (Alcatel IAP-103), 35 IAP. Very strange...

My VC self IP is 10.xx.xx.51, le Virtual IP is 10.xx.xx.151, if i look the ARP on the gateway, there is only the 51's one but the 151 is responsive... the gateway has never register the 151's ARP, and sometimes i loose the WebUI console, the 151's ping is unresponsive during 2 or 3 minutes...

Did you find the problem?

I could not find the root cause of the issue. Of course, when I got TAC on a remote session, the issue did not happen as the VC IP address became responsive after taking the master AP down. We still confirmed that the old MAC address for the former master AP was still in the ARP caches of the new master AP.

One thing that I did change before talking to TAC was upgrading to the latest code 6.5.0.0. I'm not sure what code is available since you are using the Alcatel IAP.

I would recommend trying to upgrade the code (if update is available).

Yes on Alcalel 6.5.0.0-4.3.0.1_57133 and 6.5.1.0-4.3.1.1_57902 is avaible too.

It's a critical prouduction site, so we can't upgrade without testing many RF devices before.

Hi,

Does the issue occurs when the client density is high ?

Is DPI enabled on the cluster ?

Hi,

No few clients only in this warehouse.

It happen only when the WebUI is open...

6.1.1# show dpi debug statistics

DPIMGR is not enabled

I assume DPI is not enable.

Hi,

If possible, please share the running-config from the IAP.

6.1.1# show running-config
version 6.4.4.0-4.2.4
virtual-controller-country FR
virtual-controller-key xxxxxxxxxxxxxxxxxxxxxxxxxxxx
name xx-xx-xx
organization xxxxxxx
virtual-controller-ip xx.xx.xx.xx
syslog-server xx.xx.xx.xx
terminal-access
telnet-server
ntp-server xx.xx.xx.xx
clock timezone Paris 01 00
clock summer-time CEST recurring last sunday march 00:00 last sunday october 03:00
rf-band 2.4
ams-ip xx.xx.xx.xx
ams-key xxxxxxxxxxxxxxxxxxxxxx
ams-identity xxxxxxxxxxxxxxxxxxxxxxx

allow-new-aps
allowed-ap 04:bd:88:c8:e5:a4
allowed-ap 04:bd:88:c8:e5:f2
allowed-ap 04:bd:88:c8:e6:26
allowed-ap 04:bd:88:c8:e5:c8
allowed-ap 04:bd:88:c8:e6:42
allowed-ap f0:5c:19:cc:8d:44
allowed-ap f0:5c:19:cc:8c:1c
allowed-ap f0:5c:19:cc:8d:2e
allowed-ap f0:5c:19:cc:90:c4
allowed-ap f0:5c:19:cc:90:d4
allowed-ap f0:5c:19:cc:91:02
allowed-ap f0:5c:19:cc:90:ca
allowed-ap f0:5c:19:cc:90:da
allowed-ap f0:5c:19:cc:90:ce
allowed-ap f0:5c:19:cc:90:e0
allowed-ap f0:5c:19:cc:90:9a
allowed-ap f0:5c:19:cc:90:de
allowed-ap f0:5c:19:cc:93:b6
allowed-ap f0:5c:19:cc:93:ac
allowed-ap f0:5c:19:cc:90:bc
allowed-ap f0:5c:19:cc:93:b0
allowed-ap f0:5c:19:cc:93:a6
allowed-ap f0:5c:19:cc:93:aa
allowed-ap f0:5c:19:cc:93:c8
allowed-ap f0:5c:19:cc:93:a0
allowed-ap f0:5c:19:cc:93:c6
allowed-ap f0:5c:19:cc:8e:34
allowed-ap f0:5c:19:cc:93:c2
allowed-ap f0:5c:19:cc:93:b8
allowed-ap f0:5c:19:cc:8e:2e
allowed-ap f0:5c:19:cc:8e:40
allowed-ap f0:5c:19:cc:8e:30
allowed-ap f0:5c:19:cc:8d:e8
allowed-ap f0:5c:19:cc:8e:3c
allowed-ap f0:5c:19:cc:8d:f6
allowed-ap f0:5c:19:cc:8e:2c
allowed-ap f0:5c:19:cc:8e:36
allowed-ap f0:5c:19:cc:92:ea
allowed-ap f0:5c:19:cc:8d:34
allowed-ap f0:5c:19:cc:8d:4a
allowed-ap f0:5c:19:cc:8d:10
allowed-ap f0:5c:19:cc:8d:50


snmp-server community xxxxxxxxxxxxxxxxxxxxxx

arm
 wide-bands 5ghz
 80mhz-support
 g-channels 1,5,9,13
 min-tx-power 127
 max-tx-power 127
 band-steering-mode disable
 air-time-fairness-mode fair-access
 client-aware
 scanning
 client-match
 client-match nb-matching 75

rf dot11g-radio-profile
 spectrum-monitor

rf dot11a-radio-profile
 spectrum-monitor


syslog-level warn ap-debug
syslog-level warn network
syslog-level warn security
syslog-level warn system
syslog-level warn user
syslog-level warn user-debug
syslog-level warn wireless


extended-ssid
proxy server xxxxxxxxxx.dmn01.vanilla.toplevel 8080




user 002368e4d722 xxxxxxxxxxxxxxxxxxxx portal




mgmt-user admin xxxxxxxxxxxxxxxxxxxxx


wlan access-rule default_wired_port_profile
 index 0
 rule any any match any any any permit

wlan access-rule wired-instant
 index 1
 rule masterip 0.0.0.0 match tcp 80 80 permit
 rule masterip 0.0.0.0 match tcp 4343 4343 permit
 rule any any match udp 67 68 permit
 rule any any match udp 53 53 permit

wlan access-rule BELLENUS
 index 6
 rule any any match any any any permit

wlan access-rule EDT-PAW
 index 7
 rule any any match any any any permit

wlan access-rule EDT-EMP
 index 8
 rule any any match any any any permit

wlan access-rule denyall
 index 9
 rule any any match any any any deny

wlan ssid-profile BELLENUS
 enable
 index 4
 type employee
 essid BELLENUS
 wpa-passphrase xxxxxxxxxxxxxxxxxxx
 opmode wpa2-psk-aes
 max-authentication-failures 0
 vlan xxx
 rf-band 2.4
 captive-portal disable
 dtim-period 1
 broadcast-filter arp
 deny-inter-user-bridging
 dmo-channel-utilization-threshold 90
 local-probe-req-thresh 0
 max-clients-threshold 64

wlan ssid-profile EDT-PAW
 enable
 index 5
 type employee
 essid EDT-PAW
 wpa-passphrase xxxxxxxxxxxxxxxx
 opmode wpa2-psk-aes
 max-authentication-failures 0
 vlan xxx
 rf-band all
 captive-portal disable
 hide-ssid
 dtim-period 1
 broadcast-filter arp
 deny-inter-user-bridging
 dmo-channel-utilization-threshold 90
 local-probe-req-thresh 0
 max-clients-threshold 64

wlan ssid-profile EDT-EMP
 enable
 index 6
 type employee
 essid EDT-EMP
 opmode wpa2-aes
 max-authentication-failures 0
 vlan xxx
 auth-server xxxxxxxxxx
 auth-server xxxxxxxxxx
 set-role-machine-auth denyall denyall
 rf-band all
 captive-portal disable
 hide-ssid
 dtim-period 1
 broadcast-filter arp
 deny-inter-user-bridging
 dmo-channel-utilization-threshold 90
 local-probe-req-thresh 0
 max-clients-threshold 64

auth-survivability cache-time-out 24



wlan auth-server xxxxxxx
 ip xx.xx.xx.xx
 port 1812
 acctport 1813
 key xxxxxxxxxxxxxxxx
 nas-ip xx.xx.xx.xx
 nas-id BAL-SW-WIFI

wlan auth-server xxxxxxxxxxxx
 ip xx.xx.xx.xx
 port 1812
 acctport 1813
 key xxxxxxxxxxxxxx
 nas-ip xx.xx.xx.xx
 nas-id BAL-SW-WIFI

wlan external-captive-portal
 server localhost
 port 80
 url "/"
 auth-text "Authenticated"
 auto-whitelist-disable
 https


blacklist-time 3600
auth-failure-blacklist-time 3600

ids
 wireless-containment none


wired-port-profile wired-instant
 switchport-mode access
 allowed-vlan all
 native-vlan guest
 no shutdown
 access-rule-name wired-instant
 speed auto
 duplex auto
 no poe
 type guest
 captive-portal disable
 no dot1x

wired-port-profile default_wired_port_profile
 switchport-mode trunk
 allowed-vlan all
 native-vlan 1
 shutdown
 access-rule-name default_wired_port_profile
 speed auto
 duplex full
 no poe
 type employee
 captive-portal disable
 no dot1x


enet0-port-profile default_wired_port_profile

uplink
 preemption
 enforce none
 failover-internet-pkt-lost-cnt 10
 failover-internet-pkt-send-freq 30
 failover-vpn-timeout 180


airgroup
 disable

airgroupservice airplay
 disable
 description AirPlay

airgroupservice airprint
 disable
 description AirPrint

Hi,

Thank you for sharing the log file.

I do not see any specific parameters which could be leading to this.

The config shows vlan id mapped to SSID profiles which tells me that

mgmt/client vlan are different (which is recommended)

I can suggest the following:

1. Disable background spectrum monitoring on the radios.

2. is max/min power kept at 127/127 due to any specific reason ?

ARM profile allows you to specify different transmit powers for 2.4 vs 5 Ghz

2. In case the VC becomes unresponsive, check if we get console access to it.

Run a continuous ping to VC from uplink switch:

If we get console access, collect the following:

IAP# show tech-support

IAP# show process

IAP# show cpu details

IAP# debug pkt type icmp arp

IAP# debug pkt dump

Along with that , we can collect VC uplink capture to determine the traffic

pattern.