Controllerless Networks

Hi,

I'm testing out IAP115s. I've setup 2 IAP's working together through one Virtual Controller. That works fine. We want to manage them in Airwave, that worked fine too until I did a firmware upgrade. This is what I did: I created the package in Airwave and started the upgrade of the IAP's in Airwave. That all worked out well, but after the upgrade (which was succesful: the IAP's show the correct version), I don't seem to get them in Airwave anymore. It's like Airwave "lost" them and doesn't want them anymore... I tried the following:

- changing the shared key

- deleting all SNMP settings, when I saw that didn't work, I added the v2 string again

- I rebooted the IAP's several times: after each reboot, I can connect to them as a client and everything still works, only the message that they're managed by Airwave is gone.

- in Airwave I monitored with this command: tail -f /var/log/httpd/access_log, but I don't see the IAP's connecting to Airwave.

- I can still ping Airwave from the IAP's

- no firewalls in between there

- I tried to find the group setting in Airwave, where you can configure the shared key, but I can't find it anymore. Could that be a setting that got changed in there somewhere?

Does anyone know what else I can try to solve this problem? I'm kinda out of ideas...

Oh, by the way, the upgrade was from firmware version: ArubaOS Version 6.3.1.1-4.0.0.1 (build 41049 / label #41049)

to version ArubaInstant_Pegasus_6.4.2.0-4.1.1.1_46936

Thanks!

What is your version of Airwave? 

Unless you are on 8.0.x, it is likely that you may need to upgrade your Airwave.

A few things:

Instead of the access_log, you'll want to look at var/log/pound -> this is the log that shows IAP communication to AirWave.  The access_log is supposed to be for AirWave UI session tracking, not IAP communication, so I wouldn't expect to see IAP data in that log.

Additionally, you can enable a qlog: # qlog enable swarm_debug

The output is encrypted, so you can work with support to decrypt if needed.  This helps if you end up opening up a support case.

Also, instead of manually downloading from the support site and then uploading to AirWave, if you AirWave can reach the internet, it can pull images from our image server directly.  To do this -> go to group firmware enforce -> select an image from the remote image list -> when you schedule the upgrade, AirWave will download the image from the remote server and then retain it locally.

Thanks for your replies!

@Michael_Clarke: Our version of Airwave is 8.0.5

@rgin 

in the pound log, the only line that keeps on coming back is

Dec 10 08:44:24 <our airwave DNS name> pound: <IP of a monitoring tool> GET / HTTP/1.1 - HTTP/1.1 200 OK

So apparently, the IAPs don't try to connect to Airwave. (If I go into CLI on Airwave as well as the IAP's, they can still ping each other). Is there a way (or a command) to force the IAP Virtual Controller to connect to Airwave?

Thanks for your tip on the online downloading. I'll try that the next time we have IAPs that need an OS upgrade or if I can get them back in Airwave :-)

The IAP's do keep on sending syslogs to Airwave. There they appear under the "unknown" sender. See screenshot attached.

Next step:

1) temporarily disable syslog

2) take packet capture from AirWave of https traffic

3) take packet capture from IAP of https traffic

4) might be worth it to try taking captures from your firewall of traffic to/from the IAP and to/from AirWave

Check to see which route the IAPs are trying to talk back to AirWave over.

If you need assistance with analyzing the captures, open a TAC case.

Sorry for the late reaction, I couldn't continue with this until now

(I wanted to create a new topic at first, but that didn't work very well)

To recapitulate the problem: we had 2 IAP's with 1 virtual controller. These were all managed by Airwave. Then we did an OS upgrade of the AP's (through Airwave) and since then I can't seem to get them back in Airwave. Even factory_reset them, still no joy.

Here is the AP-Debug log of an instant AP that can't connect to Airwave (doesn't appear in Airwave):

Feb 9 11:23:25 awc[1663]: awc_login: awc_init
Feb 9 11:23:25 awc[1663]: awc_init_connection: 2003: connecting to x.x.x.x:443
Feb 9 11:23:25 awc[1663]: tcp_connect: 163: recv timeout set to 5
Feb 9 11:23:25 awc[1663]: tcp_connect: 170: send timeout set to 5
Feb 9 11:23:25 awc[1663]: awc_init_connection: 2042: connected to x.x.x.x:443
Feb 9 11:23:25 awc[1663]: Failed to establish SSL connection: Error code is -1:ASN parsing error, invalid input
Feb 9 11:23:25 awc[1663]: awc_login: awc_init error
Feb 9 11:23:43 syslog: check_sid_type: sid check type, result-'0 admin'
Feb 9 11:24:14 syslog: check_sid_type: sid check type, result-'0 admin'
Feb 9 11:24:26 awc[1663]: awc_connect to x.x.x.x
Feb 9 11:24:26 awc[1663]: tcp_connect: 163: recv timeout set to 5
Feb 9 11:24:26 awc[1663]: tcp_connect: 170: send timeout set to 5
Feb 9 11:24:26 awc[1663]: awc connected to x.x.x.x
Feb 9 11:24:30 syslog: check_sid_type: sid check type, result-'0 admin'
Feb 9 11:24:30 syslog: process_msg_ref: 20: got msg_ref of len 6024 and body '/tmp/.cli_msg_UXUJEQ'
Feb 9 11:24:30 syslog: process_msg_ref: 33: opening '/tmp/.cli_msg_UXUJEQ'
Feb 9 11:24:30 syslog: process_msg_ref: 38: reading large msg
Feb 9 11:24:30 syslog: process_msg_ref: 41: read large msg of 6023 bytes
Feb 9 11:24:43 syslog: check_sid_type: sid check type, result-'0 admin'
Feb 9 11:25:14 syslog: check_sid_type: sid check type, result-'0 admin'
Feb 9 11:25:26 awc[1663]: awc_login: awc_init
Feb 9 11:25:26 awc[1663]: awc_init_connection: 2003: connecting to x.x.x.x:443
Feb 9 11:25:26 awc[1663]: tcp_connect: 163: recv timeout set to 5
Feb 9 11:25:26 awc[1663]: tcp_connect: 170: send timeout set to 5
Feb 9 11:25:26 awc[1663]: awc_init_connection: 2042: connected to x.x.x.x:443
Feb 9 11:25:26 awc[1663]: Failed to establish SSL connection: Error code is -1:ASN parsing error, invalid input
Feb 9 11:25:26 awc[1663]: awc_login: awc_init error
Feb 9 11:25:43 syslog: check_sid_type: sid check type, result-'0 admin'
Feb 9 11:26:14 syslog: check_sid_type: sid check type, result-'0 admin'

---------------------------------------

The lines in bold are something I think is linked to the OS upgrade. Or am I wrong?

Is this solveable? Do we need to do another OS upgrade/downgrade?

We upgraded our IAP115s from

ArubaOS Version 6.3.1.1-4.0.0.1 (build 41049 / label #41049)

to

ArubaInstant_Pegasus_6.4.2.0-4.1.1.1_46936

Thanks for your reply!

If you factory reset the IAPs, then they take on a new GUID that is not the same as in the database (also is the Instant 'secret' the same as before?).  I'd expect to hear connection attempts from the IAP into /var/log/pound.  But those would be ignored if the GUIDs aren't matching up.

The quick way to fix would be to try deleting the VC & IAP entries from the AirWave server (you can go to the group or folder list, click the link for modify devices to the right of the custom list view drop down, a new drop down will appear - select delete from that drop down, check the boxes for VC and APs, and then delete).

After deletion, it may take a few minutes for the next inbound message from the IAP to be received.  And depending on AMP settings, the VC entry will either return in the 'new devices' list or auto add into the group/folder (if IAP whitelisting was used).

Hi Rob,

Thanks for your quick reply!

- When I was still working on the issue (about a month ago), I already deleted the IAPs and the VC in Airwave. I even deleted the group where I put those devices in, but not the folder, because there are live devices active. Yesterday I tried the same secret, I tried another secret, I tried the Airwave settings manually within the IAPs, I tried with the DHCP options 43 and 60 - those options were correctly recognized by the IAP: I saw that throug a serial cable when it was starting up, ... 

Could it be that the secret and/or other settings are still somewhere in the config of Airwave after I deleted the group? If so, where could I find it and is it deletable?

- There is no firewall intervening: I could succesfully open an SSH session from the CLI on one of  the IAPs to Airwave (also, that subnet is known to work for Aruba devices connecting to Airwave).

- I just checked the /var/log/pound, but the IP addresses of  the IAPs don't appear there.

- We don't use the white listing.

- Our Airwave version is:  8.0.6.1

To clarify what I did with the IAP yesterday: I factory reset it and only filled in the Airwave info under system > info. Also I configured the time and our NTP server. All other settings are default from after the factory reset. I did this on one of the 2 IAPs.

I just saw that there was an OS update for the IAPs available. The IAP stated that it could download it from the internet and upgrade it itself, so I let it do that (only one of the 2). The upgrade was succesful, now the IAP OS is 6.4.2.3-4.1.1.2_48114

This sounds very strange to me.  Is there any error output in /var/log/swarm_checker or swarm_handler?

It may be worth it to open a case with support, they can double check to see if any remnants of the original VC cluster was still in the database contradicting the re-add action.

It would also be worth it to have some collection of additional swarm logs for support:

# qlog enable swarm_debug

And then to disable:

# qlog disable all

I executed the command # qlog enable swarm_debug, then I started the IAP. I left it like this for 15 minutes before executing the disable command.

In the logs, there's no useful feedback for the problem:

---

[root@airwave mercury]# qlog enable swarm_debug
[root@airwave mercury]# tail -f /var/log/swarm_checker
send_swarm_handler_command: Connection timed out at /usr/local/airwave/lib/perl/Mercury/Utility/Swarm.pm line 491.
send_swarm_handler_command: Connection timed out at /usr/local/airwave/lib/perl/Mercury/Utility/Swarm.pm line 491.

^C
[root@airwave mercury]# tail -f /var/log/swarm_handler
Mon Jan 12 11:50:52 2015: Started (PID: 19161)
Mon Jan 12 11:50:52 2015: Postgres PID: 19162
Mon Jan 12 11:57:57 2015: Started (PID: 1675)
Mon Jan 12 11:57:57 2015: Postgres PID: 1676
Wed Jan 21 15:13:02 2015: Started (PID: 15135)
Wed Jan 21 15:13:02 2015: Postgres PID: 15136
Fri Jan 23 08:37:51 2015: Started (PID: 1765)
Fri Jan 23 08:37:51 2015: Postgres PID: 1766
Thu Jan 29 04:20:30 2015: Started (PID: 32507)
Thu Jan 29 04:20:30 2015: Postgres PID: 32509
^C
[root@airwave mercury]# qlog disable all

---

Thanks for your replies, I'll contact our reseller to open a case.

The qlog outputs to /var/log/amp_diag/swarm_debug

It should give you a better idea of any swarm traffic to/from AMP.

we encountered exacty the same problem when upgrading the iap. I see the error in the iap log: Failed to establish SSL connection: Error code is -1:ASN parsing error, invalid input

Is there a solution available?