Network Management

Hi everyone,

I'm having troubles with my Airwave server. I can't perform a backup of my master controler, the automated nightly backup fail and manually "Create backup now" also doesn't work.

It seems like there's a problem with the command line because after starting the backup we see in the event log this message : "Telnet/SSH Error: command timed-out"

You can see in the next image the event log on Airwave for my master controler

The last backup correctly made by Airwave is from a older version of the controler (6.4.0.3), I've upgraded all my controlers twice since that older version and I'm now on 6.4.2.5.

Looks like telnet/ssh is failing.  Double check the credentials for the controller on the controller's manage page to make sure they are valid and try to test from AirWave CLI.

Hi,

Sorry for the late responce, holidays obliged.

I already verified the credentials on the manage - view credential page and it's the good password.

To test the login from airwave I forced a backup and while it still fail I see a succesfull login from my Airwave in my controler process log.

Here I force Airwave to backup at 9:10

Here the log on the controler

In what way can I test it from the Airwave cli ? I found the script "on_controllers" witch runs commands to all the controlers, I tested this script and there's no problem, I see the output from all my controlers including the one that failed its backup.

Thanks for the replies

I usually test my controller access from AirWave by trying to ssh/telnet to the controller from the AirWave CLI.  Typically something like this:

# ssh <user>@<controller_ip>

> en

<insert enable password>

> show run

If that's working, then it's not an issue with credentials, and possibly an issue with the AirWave user/role access.  To check this: go to AMP Setup -> users -> find your user account -> click on the role link next to your username -> then check 'Aruba Controller role. By default, all users have Aruba Controller role defaulted to 'disabled'

Is there a firewall between them? Basically Airwave pushes a command via SSH to the controller, and asks the controller to connect back to the Airwave via sftp to push its update. If you want to backup controllers to Airwave, port 22 needs to be openned up in both directions.

Hi,

Doing an ssh login from the Airwave CLI works alright althought it did ask me (root user) to saved the RSA key as it is the first login for this user.

It's not a firewall problem because there's is none between them and the others controler are not affected. The configuration is pretty much identical between each controler the main difference being that the problematic controler is the master.

It's a pretty weird problem, because I can see that the Airwave can login to the controler and in the ssh/telnet command log I also see the Airwave login to the master controler and executing the flash backup command.

I took those lines from the Airwave ssh/telnet log :

Tue Apr  7 13:21:52 2015: 
>> backup flash
previous cmd took 11.75206s

Tue Apr  7 13:22:00 2015: 
>> exit
previous cmd took 6.51034s
<< Please wait while we tar relevant files from flash...
<< Please wait while we compress the tar file...
<< File flashbackup.tar.gz created successfully on flash.
<< Please copy it out of the switch and delete it when done.
<< (sw89211) #exit

Tue Apr  7 13:22:00 2015: 
>> exit

This is the weird thing, the login is succesfull, the command is succesfull and the backup is created, but it seems like it's taking too long ! Is it possible that the command timeout is short enought that the script think its command timed out but actually it's still going on?

Thanks a lot for all your replies.

Are you able to manually push the backup file form the controller to Airwave?

Login to the controller and run this:

copy flash: flashbackup.tar.gz scp: <Airwave-ip-address> username destfilename

Try to use the root username and password

Basically when airwave backs up the controller it logs in, runs the flash backup, and then should run the copy command. From your logs, it does not look like it's even trying to copy? Does a manual copy to AMP work?

_ELiasz

I did the manual copy from the controler, as shown here, first time didn't work because the RSA changed (we changed the airwave physical server).

(sw89211) #copy flash: flashbackup.tar.gz scp: 10.1.59.151 root flashbackup_6425_15.tar.gz
Password:*********

Secure file copy:....
Error copying file:
!!REMOTE HOST IDENTIFICATION HAS CHANGED!! It is possible someone could be eavesdropping
on you right now (man-in-the-middle attack) or the RSA host key has just been changed.
To fix RSA host key please use <delete ssh-host-addr or ssh-known-hosts>
RSA host key for 10.1.59.151 has changed and you have requested strict checking.
(sw89211) #delete ssh-host-addr ?
<ssh-host-addr>         SSH Host IP A.B.C.D

(sw89211) #delete ssh-host-addr 10.1.59.151
(sw89211) #copy flash: flashbackup.tar.gz scp: 10.1.59.151 root flashbackup_6425_15.tar.gz
Password:*********

Secure file copy:....
File uploaded successfully

When I saw the error I thought that was it, that's the debug and it probably appeared when we changed server, but the last succesfull backup from Airwave is 2 weeks before we changed the server. So that was a good try, but it wasn't the real problem.

I still have the same log in the ssh command log:

Tue Apr  7 14:21:55 2015: 
>> backup flash
previous cmd took 10.73867s

Tue Apr  7 14:22:04 2015: 
>> exit
previous cmd took 9.17999s
<< Please wait while we tar relevant files from flash...
<< Please wait while we compress the tar file...
<< File flashbackup.tar.gz created successfully on flash.
<< Please copy it out of the switch and delete it when done.
<< (sw89211) #exit

Tue Apr  7 14:22:04 2015: 
>> exit

 As you said, it doesn't look like it's trying to copy!

To anyone interested, Aruba TAC fixed my problem.

Go to the communication setting and increase the telnet/ssh timeout. As shown bellow we've increase it to the maximum of 120 seconds, manual backup works fine now, we'll see if the nightly backup works tonight.