Network Management

AirWave 8.0.10 is now available for download from the support site. Link to support site

 

New in AirWave 8.0.10 is a check for bloated database tables. This check happens early in the upgrade process, and will not proceed if there are any bloated tables.

Please open a support case if you run into the following output during the upgrade:
-----
[root@localhost mercury]# start_amp_upgrade -v 8.0.20151208.1147
Directory not specified; using /root.
Upgrade script found in local cache.
Upgrade package found in local cache.
Upgrade package found in local cache.
******************************************************************

Database tables need vacuuming before you can proceed with upgrade

******************************************************************
Upgrade aborted.
Please contact Aruba Networks Support at
1-800-WiFi-LAN or support@arubanetworks.com
[root@localhost mercury]#
----- 

 

Attached are the release notes for AirWave 8.0.10.

Thnaks Rob, finally have
"AirWave now calculates the uptime of a device based on the actual uptime of the device, not just the time
that the device is connected to the AMP."

Would still love to see a time-stamp report of when a device went down and came back up.
Feature request was added to the portal a while back ;)

I just performed an upgrade from 8.0.8.1 to 8.0.10.  According to the release notes it is bundled with CentOS 6.6, but I am still seeing OS version of 6.2.  I did perform the command that was specified at the end and rebooted.  Is the 6.6 only for new installations, or did I miss a step?

New installations afaik

I tried the process in the following article and was able to vaccum database and upgrade successfully.

 

To vacuum/reindex please run the following commands on AMP CLI :

1. Disable AMP:
# amp_disable

2. Prior to running each db command below (reindex and vacuum) stop and restart the postgres database using the commands:
# service postgresql stop
# service postgresql start
This will remove any queued up nightly jobs and allow the reindex and vacuum jobs to proceed with without waiting in the queue.

2a. If you are a super user for postgres then :
#db
airwave=> reindex database airwave
airwave=> vacuum full analyze

2b. For normal root login users, run the following commands:
AMP Version < 7.3
# /opt/airwave/bin/reindexdb -d airwave -Upostgres
# /opt/airwave/bin/vacuumdb -v -f -z -Upostgres airwave
AMP Version >= 7.3
# /opt/airwave/pgsql/bin/reindexdb -d airwave -Upostgres
# /opt/airwave/pgsql/bin/vacuumdb -v -f -z -Upostgres airwave

2c. or you can also run these for normal root login users :

# psql -Upostgres airwave
psql (8.4.4)
Type "help" for help.

airwave=# reindex database airwave;
REINDEX
airwave=# vacuum full analyze;
VACUUM
airwave=#

3. Enable AMP bye :
# amp_enable

 

Ref: http://www.airheads.eu/t5/Monitoring-Management-Location/Vacuum-Reindex-in-AMP/ta-p/177274

 

The linked article on vacuuming has some AMP version specific commands.

Is that "safe" for general use?

I'd feel nervous aobut following it unless AMP-Engineers had blessed it or extended it for the current version.

 

I've got the "needs vacuuming" message from AMP_upgrade, so I'm currently awaiting a callback from TAC with instructions. (I'm also backing up my AMP database before anything happens)

The support team has a script that targets only the bloated tables.  Running vacuums on all tables is generally not needed.

There is no way to revert so have to call in, in order for TAC to run the scripts correct?

When I say revert, I don't mean, re-install with the previous version and restore backup..

The database bloat check is pre-upgrade.  So your version shouldn't have changed if the check failed.

Awesome.

I'm having the same problem:

 

"Database tables need vacuuming before you can proceed with upgrade"

 

I ran through the reindix and vacuum procedure TWICE.  Still getting the same error when attempting to run the upgrade.  I'm at 8.0.9.2 right now.

 

Anyone have any other ideas?  I will be getting in touch with Aruba support about it tomorrow, if I don't get it sorted out between now and then.  Thanks everyone :)

 

 

 

There are some tables that even after vacuumed are still large enough to block the upgrade.  The script support has will pinpoint the specific tables that block the upgrade.  Support will either help vacuum the table further or patch to bypass the check based on the list of bloated tables.

Yes, support gave me a hand with that earlier.  He just went into the upgrade script and changed the value of the bloat check limit to be slightly higher than our level of bloat.  But, I thought it was a big jump upward, and I couldn't understand the tech's explanation for why that was fine.  Why would the limit be set at aroung 1GB, and our biggest table was aroung 6GB, and so he just set the limit in the check to be 7GB?  If 6GB is perfectly fine, why would the engineers who created the check have used such a much smaller value for it?

 

Great question.

 

For some customers, the periodic vacuuming of the database was not completing due to the amount of db activity (the db vacuum process was falling behind).  This eventually caused some issues that the check is trying to pro-actively prevent.  Symptoms were large numbers of transactions waiting for db locks to write, sluggish UI, and poor db performance (lots of items pending in queue).  The initial thought was that it only happens for customers maintaining a long history, or had a lot of unique clients, but then it was observed on customers who didn't have the same scaling which prompted the decision for more aggressive vacuuming due to the lack of a distinct pattern.

Ok, hmm..

 

So, while I'm still scratching my head about database whoas..

 

I've been seeing this message on my AMP for months:

 

"Nightly maintenance failed because of unsuccessful jobs: clean_database. Contact Aruba Technical Support"

 

And of course, I've been passive-aggressively ignoring it, daring it to manifest in some discernable problem going on somewhere.  But, nope.

 

After successfully completing the upgrade to 8.0.10 (and additional kernel patches it also included) a couple of hours ago, I went ahead and, once again, ran through the reindex and vacuum procedures covered earlier in this thread.  I was hoping it would finally result in some shrinking of the size of the database(s).  But, it still doesn't appear to have.  I'm wondering if I'm going to see any difference in the results of the nightly maintenance.  I'm kind of expecting I won't, because that problem has remained outstanding through at least a couple of previous point release upgrades.  Is there anything else I can check or do proactively to try and fix, or make sure it's fixed already, this 'clean_database' routine that's been unhappy for so long?

 

-John

 

@jpjackso

 

Are there any errors in:

/var/log/nightly_maintenance

/var/log/clean_database

 

If there are, chances are that the clean database process isn't completing still.  I suggest opening a support case and providing the logs for support to help debug further.

Actually, I have done that as part of the original case, and I'm still waiting for the engineer to get back to me now with next steps.  He said he needed to work with a higher level of support first, to try and figure out what the problem is.  That was a week or so ago, at this point.  What I'm hoping he'll come back with is instructions for manually doing a definitive cleanup of the database, even if we or I have to leave it for hours or days to complete, if need be.

 

Understood.  If you happen to have a support case number, I can see about getting a status update.  And possibly help push toward a quicker resolution.

That would be so nice of you!  :)

 

Case #1816204

 

Hm... I got word from support that the issue was resolved hours ago.  Are you still experiencing the clean_database message on the homepage?

Yup, the message is still there, as usual.

 

I pinged support escalations, so hopefully we'll can get that case closed for you soon.

Rob thanks, that is much appreciated!  Are you saying you guys know the cause of that failure message, and what the fix is?  :)

 

-John

 

Not without seeing the log file.  I don't have access to see the actual support cases, but I've got some ideas for where to look and debug.  I can hide the error, but that's not the same as resolving the issue since it'll just come back again later on.