Controllerless Networks

I have a set of IAP-205s. The original, first one setup works flawlessly (Awesome!). All of the others have the same symptom. They power on ok, load the configuration from the master, go to the user login (I can't login beause the CLI isn't ready), and then after 59 seconds (measured many times), it will reset and go through the whole process again. I've gone through the manual several times, made sure that auto-join is enabled. I even tried manual join with the mac. I can see the access points coming onto the network from the master UI access point view, but it still reboots, and I can't figure out why it's doing that. Any hints?

 

[doug@r415 Downloads]$ cat /tmp/test
User: [   42.350000] asap_send_elected_master: sent successfully
[   42.368000] process `snmpd_sap' is using obsolete setsockopt SO_BSDCOMPAT
[   42.378000] wait for stm to initialize over
[   42.380000] asap_send_elected_master: sent successfully
[   56.960000] asap_send_elected_master: sent successfully
?

APBoot 1.5.3.14 (build 45815)
Built: 2014-09-05 at 11:23:04

Model: AP-20x
CPU:   BCM53011/15, revision A0
I2C:   ready
SKU:   2
OTP:   0xeca01028
Clock:
       CPU:  800 MHz
       DDR:  533 MHz
       AXI:  400 MHz
       APB:  200 MHz
       PER:  400 MHz
DRAM:  128 MB
POST1: memory passed
SF:    Detected MX25L25635E with page size  4 kB, total 32 MB
Flash: 32 MB
PCIe0: RC, link up
       dev fn venID devID class  rev    MBAR0    MBAR1    MBAR2    MBAR3
       00  00  14e4  4360 00002   03 08000004 00000000 00000000 00000000
PCIe1: RC, link up
       dev fn venID devID class  rev    MBAR0    MBAR1    MBAR2    MBAR3
       00  00  14e4  4360 00002   03 40000004 00000000 00000000 00000000
Power: POE
Net:   eth0
Radio: bcm43460#0, bcm43460#1

Hit <Enter> to stop autoboot:  0
Booting OS partition 1
Checking image @ 0x1ef00000
Copying image from 0x1ef00000

Image is signed; verifying checksum... passed
Signer Cert OK
Policy Cert OK
RSA signature verified.
Uncompressing... done
[    0.000000]
[    0.000000] Aruba Networks
[    0.000000] ArubaOS Version 6.4.2.6-4.1.1.11 (build 52666 / label #52666)
[    0.000000] Built by p4build@chios on 2015-11-22 at 16:25:55 PST (gcc version 4.7.2)
[    0.000000]
[    0.000000] Flash variant: variant0
[    0.000000] MPCORE found at 19020000, SCTLR 0x10c53c7f ACTLR 0x00000040 CPSR 0x600001d3 SPSR 0x00024418 PRRR 0x000a81a8 NMRR 0x40e040e0
[    0.000000] Memory: 128MB = 128MB total
[    0.000000] Memory: 117680k/117680k available, 13392k reserved, 0K highmem
[    0.052000] hw perfevents: enabled with ARMv7 Cortex-A9 PMU driver, 7 counters available
[    5.270000] PMU: registered new PMU device of type 0
[    5.776000] pci 0001:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
[    5.778000] pci 0001:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
[    5.780000] pci 0001:01:00.0: BAR 0: set to [mem 0x08000000-0x08007fff 64bit] (PCI address [0x8000000-0x8007fff]
[    6.282000] pci 0002:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
[    6.284000] pci 0002:02:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
[    6.286000] pci 0002:02:00.0: BAR 0: set to [mem 0x40000000-0x40007fff 64bit] (PCI address [0x40000000-0x40007fff]
[    6.630000] i2c-springbank: using remapped base c8914060
[    6.632000] mpcore_wdt mpcore_wdt: enabling watchdog.
[    6.636000] qspi_iproc qspi_iproc.1: 1-lane output, 3-byte address
[    6.638000] m25p80 spi1.0: mx25l25635e (32768 Kbytes)
[    6.640000] Creating 1 MTD partitions on "spi1.0":
[    6.642000] 0x000000000000-0x000002000000 : "qspi"
[    6.646000]
[    6.646000] Starting Kernel SHA1 KAT ...Completed Kernel SHA1 KAT
[    6.650000] Starting Kernel HMAC-SHA1 KAT ...Completed Kernel HMAC-SHA1 KAT
[    6.662000] Starting Kernel DES KAT ...Completed Kernel DES KAT
[    6.666000] Starting Kernel AES KAT ...Completed Kernel AES KAT
[    6.668000]
Domain Name: instant.arubanetworks.com
No panic info available
apfcutil: sector FAT: Cache uninitialized
Backup ENV.
Cfg len is 3.4k
Loading configuration file of length 3.4k...
wifi uplink not present...
extended ssid config detected...
Terminal access enabled...
Valid SSID detected...
[   13.938000] ADDRCONF(NETDEV_UP): bond0: link is not ready
[   15.942000] bond0: link up (1000FD)
[   15.944000] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
Mesh is DISABLED on this device.
Disabling wpa_supplicant for this paltform
[   17.014000] AP xml model 81, num_radios 2 (jiffies 8507)
[   17.018000] init_asap_mod: installation:0
[   17.020000] radio 0: band 1 ant 0 max_ssid 16
[   17.022000] radio 1: band 0 ant 0 max_ssid 16
[   17.024000] election init: rand=13 HZ=500
[   17.026000] IAP client match init
notify asap_mod 3g no present...
Starting watchdog process...
Got all network params from APboot env. Skipping DHCP
10.0.2.23 255.255.255.0 10.0.2.1
Compressing all files in the /etc/httpd directory...
[   22.032000] ip_time_handler: Got ip and packets on bond0 Started master election 43-0, rand 33
Done.
Starting Webserver
bind: Transport endpoint is not connected
bind: Transport endpoint is not connected
bind: Transport endpoint is not connected
NTP server pool.ntp.org from configuration.
Launching NTP Client on pool.ntp.org
[   22.956000] Adding bridge entry for magic vlan GW, ifindex 8450, 40:e3:d6:ce:8f:2e
[   22.958000] (00:00:24) !!! Init ---> Slave
[   22.974000] wifi0: AP type AP-205, radio 0, max_bssids 16
[   23.018000] wifi1: AP type AP-205, radio 1, max_bssids 16
shutting down watchdog process (nanny will restart it)...
Completed SW FIPS KAT test

        <<<<<       Welcome to the Access Point     >>>>>
User:

 

<repeat>
User:

Login, and execute "show log system" to see why it rebooted.

Oops, I wasn't aware that a reply to the email wouldn't work at first.

 

The master controller logs (2 events here) don't appear to be useful:

Mar 31 11:10:20  cli[1910]: <341097> <WARN> |AP 40:e3:d6:ce:8f:2e@10.0.2.11 cli|  recv_image_check: mac-84:d4:7e:c4:15:e0, version-6.4.2.6-4.1.1.11_52666, ccode_idx-0.
Mar 31 11:10:20  cli[1910]: <341165> <WARN> |AP 40:e3:d6:ce:8f:2e@10.0.2.11 cli|  send image match to 10.0.2.23.
Mar 31 11:10:25  cli[1910]: <341199> <WARN> |AP 40:e3:d6:ce:8f:2e@10.0.2.11 cli|  send_config_init: send config to slave 10.0.2.23, using url 0, auto save disable 0.
Mar 31 11:12:40  cli[1910]: <341097> <WARN> |AP 40:e3:d6:ce:8f:2e@10.0.2.11 cli|  recv_image_check: mac-84:d4:7e:c4:15:e0, version-6.4.2.6-4.1.1.11_52666, ccode_idx-0.
Mar 31 11:12:40  cli[1910]: <341165> <WARN> |AP 40:e3:d6:ce:8f:2e@10.0.2.11 cli|  send image match to 10.0.2.23.
Mar 31 11:12:45  cli[1910]: <341199> <WARN> |AP 40:e3:d6:ce:8f:2e@10.0.2.11 cli|  send_config_init: send config to slave 10.0.2.23, using url 0, auto save disable 0.

 

 

And I cannot login to the slave as it is booting, because it says CLI is not ready up until the second it resets.

Do you only have IAP205 connected?

Sent from Outlook for iPhone

Victor: connected to what?

 

All clients are connected to the first working IAP. the other IAPs are on the same vlan. I have 2 of the 3 others plugged in to POE uplink in the same vlan, but it doesn't matter whether I have 1 or 2, they continuously reboot.

 

To the same VLAN / Cluster

Sent from Outlook for iPhone

Do you have a virtual IP assigned ?

Sent from Outlook for iPhone

I have tried with the virtual controller-IP set to the IP address of the working (1st) station. With or without that set I see the same thing.

 

(PS - tell D Stillman I said hi, if you know him)

 

Type "show version" on the commandline of the device that rebooted to see why it rebooted.

 

Colin, as I indicated in my earlier reply, the AP reboots before the CLI is ready to accept logins (59 seconds). There is no way to run any command at all since I cannot login via serial connection before it resets.

 

 

Are these new?  You should open a TAC case..

 

Hi Doug,

 

Does any of these non-working IAP's are 205-H?

 

The question might not be relevant to this issue. But I wanted to check as IAP-205 are supported on

4.1 release whereas IAP-205H are only supported from 4.2.

New: yes, though I got them off Amazon new. Working on entitlement/support registration now. I was hoping that somebody else might have seen something similar to this. I looked through the forums and see a few other issues about continuous rebooting, but wasn't able to find anything like this exactly (that had a working solution)

 

Version: here's the relevant info for one of the units currently in a reboot loop (none are 205H). They boot up and contact the master just fine, run for 59 seconds, then reboot (before CLI i ready). They also manage to get the new code from the master into Partition 1.

 

 

apboot> mfginfo
Inventory:
Card 0: System
        Wired MAC           : 84:d4:7e:c4:15:e0
        Wired MAC Count     : 2
        Date Code           : 102915
        Serial              : CM0624875
        Country             : CCODE-US-c3d8e0aa31b3d54d2674f7f45a339af06d31d135
Card 1: CPU
        Assembly            : 2010202D
        Serial              : TB5AL10B8
        Date Code           : 102815
        Major Rev           : 03
        Minor Rev/Variant   : 01

 

apboot> osinfo
Partition 0:
    image type: 0
  machine type: 30
          size: 10085068
       version: 6.4.2.6-4.1.1.6
  build string: ArubaOS version 6.4.2.6-4.1.1.6 for Taurus (p4build@tortuga) (gcc version 4.7.2) #50009 Tue May 12 10:27:51 PDT 2015
         flags: Instant preserve

Image is signed; verifying checksum... passed
Signer Cert OK
Policy Cert OK
RSA signature verified.

 

Partition 1:
    image type: 0
  machine type: 30
          size: 10100796
       version: 6.4.2.6-4.1.1.11
  build string: ArubaOS version 6.4.2.6-4.1.1.11 for Taurus (p4build@chios) (gcc version 4.7.2) #52666 Sun Nov 22 16:25:55 PST 2015
         flags: Instant preserve

Image is signed; verifying checksum... passed
Signer Cert OK
Policy Cert OK
RSA signature verified.

Hi Doug,

 

1. Is it possible for you power on one of these non-working IAP's using POE injector or adaptor & see the behavior.?

 

2. Did you had the chance to try factory reset on one of the non-working IAP's & verify the behavior?

 

apboot>factory_reset

 

In case, they still do not work, please open TAC case.

 

 

re: POE - same thing across 2 different switches (HP/Brocade) - so I'm pretty sure this is not POE. We also have countless phones, cameras, wireless antennas, and other POE devices on both.

 

re: factory reset, yes. still same. Could it be something about the specific config that is causing this? (a little bit of my added text all the way at the bottom)

 

 

apboot> factory_reset
Clearing state... Checking OS image and flags

Image is signed; verifying checksum...

Preserving image partition 0
Erasing flash sector @ 0x1ff90000....done
Erased 1 sectors

Erasing flash sector @ 0x1ffb0000....done
Erased 1 sectors

Erasing flash sector @ 0x1ffc0000....done
Erased 1 sectors

Erasing flash sector @ 0x1ffd0000....done
Erased 1 sectors

done
Purging environment... preserving os_partition (1)
Erasing flash...Writing to flash..................done
Erasing flash...Writing to flash..................done
done
apboot> boot

Booting OS partition 1
Checking image @ 0x1ef00000
Copying image from 0x1ef00000

Image is signed; verifying checksum...

Signer Cert OK
Policy Cert OK
RSA signature verified.
Uncompressing... done
[    0.000000]
[    0.000000] Aruba Networks
[    0.000000] ArubaOS Version 6.4.2.6-4.1.1.11 (build 52666 / label #52666)
[    0.000000] Built by p4build@chios on 2015-11-22 at 16:25:55 PST (gcc version 4.7.2)
[    0.000000]
[    0.000000] Flash variant: variant0
[    0.000000] MPCORE found at 19020000, SCTLR 0x10c53c7f ACTLR 0x00000040 CPSR 0x600001d3 SPSR 0x02004058 PRRR 0x000a81a8 NMRR 0x40e040e0
[    0.000000] Memory: 128MB = 128MB total
[    0.000000] Memory: 117680k/117680k available, 13392k reserved, 0K highmem
[    0.052000] hw perfevents: enabled with ARMv7 Cortex-A9 PMU driver, 7 counters available
[    5.328000] PMU: registered new PMU device of type 0
[    5.834000] pci 0001:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
[    5.836000] pci 0001:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
[    5.838000] pci 0001:01:00.0: BAR 0: set to [mem 0x08000000-0x08007fff 64bit] (PCI address [0x8000000-0x8007fff]
[    6.340000] pci 0002:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
[    6.342000] pci 0002:02:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
[    6.344000] pci 0002:02:00.0: BAR 0: set to [mem 0x40000000-0x40007fff 64bit] (PCI address [0x40000000-0x40007fff]
[    6.688000] i2c-springbank: using remapped base c8914060
[    6.692000] mpcore_wdt mpcore_wdt: enabling watchdog.
[    6.696000] qspi_iproc qspi_iproc.1: 1-lane output, 3-byte address
[    6.698000] m25p80 spi1.0: mx25l25635e (32768 Kbytes)
[    6.700000] Creating 1 MTD partitions on "spi1.0":
[    6.702000] 0x000000000000-0x000002000000 : "qspi"
[    6.706000]
[    6.706000] Starting Kernel SHA1 KAT ...Completed Kernel SHA1 KAT
[    6.710000] Starting Kernel HMAC-SHA1 KAT ...Completed Kernel HMAC-SHA1 KAT
[    6.722000] Starting Kernel DES KAT ...Completed Kernel DES KAT
[    6.726000] Starting Kernel AES KAT ...Completed Kernel AES KAT
[    6.728000]
Domain Name: instant.arubanetworks.com
No panic info available
apfcutil: sector CACHE: Cache uninitialized
apfcutil: sector RAP: Cache uninitialized
apfcutil: sector FAT: Cache uninitialized
apfcutil: sector SCRATCH: Cache uninitialized
Check backup ENV.
apfcutil: sector MESH Prov: Cache uninitialized
Cfg len is 0
Configuration file is empty, turning on default services...
extended ssid activate due to no AP config ...
[   14.610000] ADDRCONF(NETDEV_UP): bond0: link is not ready
[   16.614000] bond0: link up (1000FD)
[   16.616000] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
Mesh is DISABLED on this device.
Disabling wpa_supplicant for this paltform
[   17.684000] AP xml model 81, num_radios 2 (jiffies 8842)
[   17.686000] init_asap_mod: installation:0
[   17.688000] radio 0: band 1 ant 0 max_ssid 16
[   17.690000] radio 1: band 0 ant 0 max_ssid 16
[   17.692000] election init: rand=15 HZ=500
[   17.694000] IAP client match init
notify asap_mod 3g no present...
Starting watchdog process...
Getting an IP address...
Dec 31 16:00:19 udhcpc[1385]: udhcpc (v0.9.9-pre) started
Dec 31 16:00:19 udhcpc[1385]: send_discover: pkt num 0, secs 0
Dec 31 16:00:19 udhcpc[1385]: Sending discover...
Dec 31 16:00:19 udhcpc[1385]: send_selecting: pkt num 0, secs 0
Dec 31 16:00:19 udhcpc[1385]: Sending select for 10.0.2.23...
Dec 31 16:00:19 udhcpc[1385]: Lease of 10.0.2.23 obtained, lease time 3600
Dec 31 16:00:19 udhcpc[1385]: DHCP OPT 43 deleted airwave config

10.0.2.23 255.255.255.0 10.0.2.1
Compressing all files in the /etc/httpd directory...
[   22.704000] ip_time_handler: Got ip and packets on bond0 Started master election 12-0, rand 29
[   23.124000] Adding bridge entry for magic vlan GW, ifindex 8450, 40:e3:d6:ce:8f:2e
[   23.126000] (00:00:25) !!! Init ---> Slave
Done.
Starting Webserver
bind: Transport endpoint is not connected
bind: Transport endpoint is not connected
bind: Transport endpoint is not connected
Launching NTP Client on pool.ntp.org
[   24.696000] wifi0: AP type AP-205, radio 0, max_bssids 16
[   24.740000] wifi1: AP type AP-205, radio 1, max_bssids 16
shutting down watchdog process (nanny will restart it)...
 
        <<<<<       Welcome to the Access Point     >>>>>
Completed SW FIPS KAT test
 
User: admi[   44.354000] asap_send_elected_master: sent successfully
[   44.374000] process `snmpd_sap' is using obsolete setsockopt SO_BSDCOMPAT
n
System uptime is 46 seconds and CLI is not ready yet, please try again later.
User: [   44.722000] wait for stm to initialize over
[   44.724000] asap_send_elected_master: sent successfully
admin
System uptime is 53 seconds and CLI is not ready yet, please try again later.
User: admin
System uptime is 56 seconds and CLI is not ready yet, please try again later.
User: admin
System uptime is 60 seconds and CLI is not ready yet, please try again later.
User: [   60.094000] asap_send_elected_master: sent successfully

<reset here>

APBoot 1.5.3.14 (build 45815)
Built: 2014-09-05 at 11:23:04

Model: AP-20x
CPU:   BCM53011/15, revision A0
I2C:   ready

 

 

So, this is the first time I've actually seen it hit 60 seconds. It appears there is some internal watchdog timer or something that is hitting at exactly 60 seconds, but I can't figure out what.

 

 

Please open a TAC case.

Hi,

 

is there any solution or Idea yet?

 

I'm also currently hitting (and hunting) some similar symptoms and what I just stumbled over:

It seems that the latest IAP software releases require a reachable NTP server to boot up proplery.

And as your IAP is saying a second timestamp:

 

 

User: [   60.094000] asap_send_elected_master: sent successfully

 

 

it might be that it is also not able to fetch a proper time from an NTP server.

 

Was it the NTP time issue? Did TAC find a solution?

 

Thanks & Bye,

     Chris

 

 

My case is still open. This seems to be isolated (for me) to my Brocade switches. if I plug the IAP-205 into an HP switch it is working fine. Very odd. My next step is to collect trace/mirror data from the brocade, but your theory is interesting. In my case, it's exactly the same DHCP config no matter which switch I plug it into.

Meanwhile, I have plugged a Dell branded IAP-205 into the same port on the same Brocade and it comes up fine, but then I can't login to it because of an Internal Error 8, which is a different issue.

Hi DougDoug,

 

I know TAC tickets currently take very long, so I'm also searching the forum for odd things before contacting TAC.

 

I might have an other hint for you: (While building my survey eqipment) I also stumbled over a quite odd behaviour for IAP's - When it is the only device on an switch it doesn't go into the election mode properly. It needs some 'real' traffic to start the election mode - I had to connect a RasperryPI to the switch that just ping's the IAP's IP and then the IAP started working properly...

 

But when i remember right the key was the message

[   22.704000] ip_time_handler: Got ip and packets on bond0 Started master election 12-0, rand 29

which is here in your case...

 

odd thing. Maybe the brocade is blocking some of the Layer2 broadcast traffic flowing between the IAP's?

Do you have an ethertap or a similar box to capture the l2 traffic between the switch and iap? mabye this gives you an idea ;)

 

Bye,

     Chris

 

I was going to setup port mirroring, but once I got the IAP working on
the HP (which is connected through the brocade to another HP where the
master IAP-205 lives), everything started working. It's all a bit strange.

Also, it changed from 60 seconds to reboot to 30 seconds to reboot last
week. But for now, I have users actually using it connected to the HP,
so I'm a bit loathe to change it. The Dell IAP-205, which should have
exactly the same hardware and firmware, gives me the "Internal error 8"
message, but refuses login or to turn on its radio after that. It
doesn't reboot, though.

I need to get my second machine working in there for the port mirroring
tap (days - needs a new CPU) or to take the laptop in with wireshark,
turn on mirroring, move the AP, and see what happens and send it in,
soonish, but probably not today.

That ping thing is interesting. There'd be the same or less traffic on
the HP switch where it is currently working. Aruba tech had a loose
theory that it could be a POE negotiation thing, but the exactness of
the timing makes me doubt this.

answer:

in earlier version of the OS, it was allowed to run an IAP-2015 in class 2 POE and it would work but disable features. The unit would continue to run just fine, but not full functionality.

In newer releases of AirOS (starting about March/April 2016), class 2 POE is no longer sufficient even to complete a boot cycle. It will exceed the power usage of the port which will disable POE and the AP will reboot, thus the loop. THe weird thing was that it always happened at *exactly* the same time. So, there's something very deterministic in the boot sequence that enables a particular feature that just tips it over the edge, the POE shuts off temporarily on overload, and the switch reboots.

Answer: minimum requirement for IAP-205 is class 3 POE. The documentation is being adjusted going forward.

 

(long overdue followup)

VIctor, after re-reading, I now think you are asking if I have any other model of Aruba access point. No. all same.