Network Management

So, here's the output I got from trying to upgrade last night:

[root@spipair01 mercury]# start_amp_upgrade -v 8.0.10
Directory not specified; using /root.
Upgrade script found in local cache.
Upgrade package found in local cache.
Upgrade package found in local cache.
Checking for any patches or modifications to the code
This process may take a few minutes...
Modified files:
? lib/html/static/Content
? lib/html/static/Data
? lib/html/static/Resources
? lib/html/static/Skins
? lib/html/static/SupportedDevices.htm
? lib/html/static/SupportedDevices.js
? lib/html/static/SupportedDevices.mcwebhelp
? lib/html/static/SupportedDevicesArubaHTML5.mclog
? lib/html/static/SupportedDevices_CSH.htm
? lib/html/static/csh.js
WARNING: AMP contains patches or modifications to the code
These changes are likely to be overwritten during the upgrade.
For additional information please contact Aruba Networks Support at:
1-800-WiFi-LAN or support@arubanetworks.com
Would you like to continue with the upgrade? (yes/no): yes
Continuing with upgrade.
Patches and modifications are likely to be overwritten.
Validating the upgrade package...
Upgrade package is OK.
Upgrading AMP to version 8.0.10 from version 8.0.9.2...
Detailed log will be written to /var/log/upgrade/AMP-8.0.10-upgrade.log
STEP 1: Moving old version aside.
STEP 2: Unpacking upgrade package.
STEP 3: Checking for compatibility.
STEP 4: Stopping AMP services.
STEP 5: Installing upgrade.
tsdb-server: unrecognized service
make[3]: *** [gnutls] Error 3
make[3]: Leaving directory `/root/svn/mercury/src/x86_64/rpms/CentOS-6'
make[2]: *** [perllib] Error 2
make[2]: Leaving directory `/root/svn/mercury/src'
make[1]: *** [src_install] Error 2
make[1]: Leaving directory `/root/svn/mercury'
make: *** [upgrade] Error 2
make: Leaving directory `/root/svn/mercury'
Stopping httpd: [ OK ]
tsdb-server: unrecognized service

Upgrade aborted.
Please contact Aruba Networks Support at
1-800-WiFi-LAN or support@arubanetworks.com
DISABLED - 2!!
*** WARNING *** Directory may have been renamed out from under you: /bin/pwd=/root/svn_old/mercury
[root@spipair01 mercury]#

 

 

I was able to successfully revert back to 8.0.9.2:

 

[root@spipair01 mercury]# start_amp_upgrade -v 8.0.9.2
Directory not specified; using /root.
Upgrade script found in local cache.
Upgrade package found in local cache.
Upgrade package found in local cache.
Checking for any patches or modifications to the code
This process may take a few minutes...
Modified files:

?       lib/html/static/Content

?       lib/html/static/Data

?       lib/html/static/Resources

?       lib/html/static/Skins

?       lib/html/static/SupportedDevices.htm

?       lib/html/static/SupportedDevices.js

?       lib/html/static/SupportedDevices.mcwebhelp

?       lib/html/static/SupportedDevicesArubaHTML5.mclog

?       lib/html/static/SupportedDevices_CSH.htm

?       lib/html/static/csh.js

WARNING: AMP contains patches or modifications to the code
These changes are likely to be overwritten during the upgrade.
For additional information please contact Aruba Networks Support at:
1-800-WiFi-LAN or support@arubanetworks.com
Would you like to continue with the upgrade? (yes/no): yes
Continuing with upgrade.
Patches and modifications are likely to be overwritten.
WARNING: There is a pre-upgrade backup from a previous attempt to upgrade
to this version.
Would you like to overwrite it with the latest nightly backup? (yes/no): yes
Validating the upgrade package...
Upgrade package is OK.
Upgrading AMP to version 8.0.9.2 from version 8.0.9.2...
Detailed log will be written to /var/log/upgrade/AMP-8.0.9.2-upgrade.log

STEP 1: Moving old version aside.

STEP 2: Unpacking upgrade package.

STEP 3: Checking for compatibility.

STEP 4: Stopping AMP services.

STEP 5: Installing upgrade.







Duplicate devices or devices with incomplete information exist in the AMP database.
See /var/log/duplicate_or_incomplete_devices for details.

For additional information please contact Aruba Networks Support at:
1-800-WiFi-LAN or support@arubanetworks.com

STEP 6: Restarting AMP services.

What do I need to do before I re-attempt?

 

Thanks,

 

Russell

Wish you hadn't done that 'revert'.  Downgrading isn't supported, and that may have made things worse since the initial upgrade may have already updated some DB schema and OS rpms.

 

I would suggest grabbing an old nightly backup and pulling it off server just in case, and then open a support case to see if they can help get you back to a good state.  Include /var/log/upgrade/AMP-8.0.10-upgrade.log  (only the last 100 lines are really needed, but if you open a support case, it's easier to just give the full log).

Rob is 100% correct.
Also, i can't find the previous post to this but you could always try the following once you get that error.
Restore a previous backup and try the upgrade again and if the error happens, run the commands below.

Rob, you posted this somewhere sometime ago.
This should do the trick
# yum install glibc
Then double check to verify the rpm package is there:
# rpm -qa | grep -i glibc

@Pasquale

 

True the glibc package worked in a previous upgrade.  But this current error may be due to a different RPM package.  The most recent RPMs touched in the codebase are hints that it could be related to either postgresql or gnutils.  We'd know for sure with the output from the upgrade log.

I may have put myself in a bind by doing the reversion, since the log file of the failed upgrade no longer exists in /var/log/upgrade. Unless there is somewhere else to look? I tried the upgrade/verification on glibc on the night, as that was a thread I found, but to no avail. Given that I may just need to try again and try to push through it, is there something I need to look for in the log that may tell me which RPM may be missing/outdated?

 

Russell

 

If you want to get to the best possible state, I would suggest starting from the last known good backup.  To do this, you'd have to grab the last known backup from /var/airwave-backup that happened before the upgrade (it may be the pre-upgrade backup if the original backups are aged out), also make sure you save a copy of the license key from Home -> License, then reinstall 8.0.9.2 and restore the backup.  This will ensure that the data is in a good and sane state.

 

When you try the upgrade again, here are the key phrase patterns to look out for in the /var/log/upgrade log:

1) package <some rpm> is already installed

2) error: Failed dependencies:  <and then the list of the fails>

 

If you decide to proceed from the current 'reverted' state, please be aware that there may be some unforeseen issues since we don't support roll-back/reverts/downgrades of AMP code.

Russell, I think I've run into the same or similar issue as you did.  Curious here as to how it was ultimatly resolved? 

 

[root@amp mercury]# start_amp_upgrade -v 8.0.10
Directory not specified; using /root.
Upgrade script found in local cache.
Upgrade package found in local cache.
Upgrade package found in local cache.
Checking for any patches or modifications to the code
This process may take a few minutes...
Modified files:

?       lib/html/static/Content

?       lib/html/static/Data

?       lib/html/static/Resources

?       lib/html/static/Skins

?       lib/html/static/SupportedDevices.htm

?       lib/html/static/SupportedDevices.js

?       lib/html/static/SupportedDevices.mcwebhelp

?       lib/html/static/SupportedDevicesArubaHTML5.mclog

?       lib/html/static/SupportedDevices_CSH.htm

?       lib/html/static/csh.js

WARNING: AMP contains patches or modifications to the code
These changes are likely to be overwritten during the upgrade.
For additional information please contact Aruba Networks Support at:
1-800-WiFi-LAN or support@arubanetworks.com
Would you like to continue with the upgrade? (yes/no): yes
Continuing with upgrade.
Patches and modifications are likely to be overwritten.
Validating the upgrade package...
Upgrade package is OK.
Upgrading AMP to version 8.0.10 from version 8.0.9.2...
Detailed log will be written to /var/log/upgrade/AMP-8.0.10-upgrade.log

STEP 1: Moving old version aside.

STEP 2: Unpacking upgrade package.

STEP 3: Checking for compatibility.

STEP 4: Stopping AMP services.

STEP 5: Installing upgrade.
tsdb-server: unrecognized service
make[3]: *** [gnutls] Error 3
make[3]: Leaving directory `/root/svn/mercury/src/x86_64/rpms/CentOS-6'
make[2]: *** [perllib] Error 2
make[2]: Leaving directory `/root/svn/mercury/src'
make[1]: *** [src_install] Error 2
make[1]: Leaving directory `/root/svn/mercury'
make: *** [upgrade] Error 2
make: Leaving directory `/root/svn/mercury'
Stopping httpd:                                            [  OK  ]
tsdb-server: unrecognized service

Upgrade aborted.
Please contact Aruba Networks Support at
1-800-WiFi-LAN or support@arubanetworks.com
DISABLED - 2!!
*** WARNING *** Directory may have been renamed out from under you: /bin/pwd=/root/svn_old/mercury
[root@amp mercury]#

 

 

Where it appears to to go south is here:

 

make[2]: Entering directory `/root/svn/mercury/src'
/usr/bin/make -f Makefile.perllib remove_old_rpms
make[3]: Entering directory `/root/svn/mercury/src'
make[3]: Leaving directory `/root/svn/mercury/src'
/usr/bin/make -C /root/svn/mercury/src/x86_64/rpms/CentOS-6 install
make[3]: Entering directory `/root/svn/mercury/src/x86_64/rpms/CentOS-6'
Preparing...                ##################################################
 package gnutls-2.8.5-19.el6_7.x86_64 (which is newer than gnutls-2.8.5-18.el6.x86_64) is already installed
make[3]: *** [gnutls] Error 3
make[3]: Leaving directory `/root/svn/mercury/src/x86_64/rpms/CentOS-6'
make[2]: *** [perllib] Error 2
make[2]: Leaving directory `/root/svn/mercury/src'
make[1]: *** [src_install] Error 2
make[1]: Leaving directory `/root/svn/mercury'
make: *** [upgrade] Error 2
make: Leaving directory `/root/svn/mercury'
Stopping httpd: [  OK  ]
tsdb-server: unrecognized service

 

I've not attempted any recovery ops yet, apart from snagging the autobackup archive from the previous four evenings.  Any tricks that may allow a lowly sysop to fix this would be appreciated.

 

 

 

For tracking purposes, this is DE23915.  The main issue is gnutls rpm version mismatch is too strict (the version present is newer than expected).  The fix is will be included in the next 8.0 release, but in the meantime:

 

IF /root/svn/mercury VERSION IS 8.0.10 (SEE URL LINE)

[root@amp mercury]# pwd
/root/svn/mercury

[root@amp mercury]# svn info
Path: .
Working Copy Root Path: /root/svn/mercury
URL: http://svn.corp.airwave.com/usr/local/svnroot/mercury/tags/RELEASE_8_0_10
Repository Root: http://svn.corp.airwave.com/usr/local/svnroot/mercury
Repository UUID: 99f6a4a8-5413-0410-a0fb-e6b940866760
Revision: 96391
Node Kind: directory
Schedule: normal
Last Changed Author: rgin
Last Changed Rev: 96390
Last Changed Date: 2015-12-08 16:37:51 -0500 (Tue, 08 Dec 2015)

 

AND IF RPM Version:

[root@amp mercury]# rpm -qa | grep -i gnutl
gnutls-2.8.5-19.el6_7.x86_64

if it's not this version or 2.8.5-18, then you'll want to #yum update gnutls

 

THEN, you have 2 options.

1) patch and apply code (steps forthcoming)

2) open a support case and then they will perform item #1

 

######################################################################

############                                  ITEM 1 STEPS                                  ############

######################################################################

 

A) create a patch file in /var/airwave/custom

# vi /var/airwave/custom/upgradefix.diff

(((copy everything from 'Index' to the line before ###PATCH END###)))

 

Index: src/x86_64/rpms/CentOS-6/Makefile
===================================================================
--- src/x86_64/rpms/CentOS-6/Makefile (revision 96391)
+++ src/x86_64/rpms/CentOS-6/Makefile (working copy)
@@ -10,7 +10,7 @@
INSTALLRPM = ../../../install_rpm
RPM_AT_LEAST = ../../../../bin/rpm_at_least

-install: security_patches gnutls dracut tzdata pdfsupport redis tsdb svn bfafw node performance
+install: security_patches dracut tzdata pdfsupport redis tsdb svn bfafw node performance

security_patches:

@@ -193,12 +193,6 @@
$(INSTALLRPM) gnupg2-2.0.14-6.el6_4.x86_64.rpm; \
fi

-# gnutls-utils provides certtool, which is part of our CAC auth implementation
-gnutls:
- @if ! $(RPM_AT_LEAST) gnutls-utils 2.8.5-18.el6; then \
- $(INSTALLRPM) gnutls-utils-2.8.5-18.el6 gnutls-utils-2.8.5-18.el6.x86_64.rpm gnutls-2.8.5-18.el6.x86_64.rpm ; \
- fi
-
# dracut rpms were added as a dependency for kernel updates
dracut:
@if ! $(RPM_AT_LEAST) dracut 004-388.el6; then \

###PATCH END###  (THIS LINE NOT NEEDED IN PATCH)

###PATCH END###  (THIS LINE NOT NEEDED IN PATCH)

 

 

B)  Test that the newly created patch will apply cleanly (this is just good practice)

# cd /root/svn/mercury

# patch -p0 --dry-run < /var/airwave/custom/upgradefix.diff

If the 2 hunks in the patch do not apply cleanly, then you'll want to open a support case to have support take a look.

 

C) Apply the patch

# patch -p0 < /var/airwave/custom/upgradefix.diff

# make

 

D) Enable AMP

# amp_enable -now

 

E) Wait 5-10 minutes for AMP processes to start up

If you'd like to monitor status of AMP coming online

#watch daemons

OR

#wd

Both commands do the same thing.  When you see about a dozen processes listed, the UI should be reachable.

 

 

 

 

Almost forgot, if you decide to create the patch on Windows and then move it over using SCP, you may get the mysterious hidden characters that need to be removed.  If so:

 

# dos2unix /var/airwave/custom/upgradefix.diff

There's a possibility that your AMP may not have the dos2unix package, if so, you can yum install dos2unix, or find and install dos2unix-3.1-37.el6.x86_64.rpm.  Once you do that, i don't expect any issues from my workaround steps.

 

since 8.0.11 is out now, can I just upgrade directly to that, or should I go through all of the steps above to get to 8.0.10 first?

 

Russell

You should be able to go directly to 8.0.11 w/o needing to perform the workaround steps.