Network Management

last person joined: 6 hours ago 

Keep an informative eye on your network with IMC and AirWave network management solutions.
Expand all | Collapse all

AirWave 8.0.10 now available. Upgrade has a database bloat check.

  • 1.  AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Dec 09, 2015 07:03 PM
      |   view attached

    AirWave 8.0.10 is now available for download from the support site. Link to support site

     

    New in AirWave 8.0.10 is a check for bloated database tables. This check happens early in the upgrade process, and will not proceed if there are any bloated tables.

    Please open a support case if you run into the following output during the upgrade:
    -----
    [root@localhost mercury]# start_amp_upgrade -v 8.0.20151208.1147
    Directory not specified; using /root.
    Upgrade script found in local cache.
    Upgrade package found in local cache.
    Upgrade package found in local cache.
    ******************************************************************

    Database tables need vacuuming before you can proceed with upgrade

    ******************************************************************
    Upgrade aborted.
    Please contact Aruba Networks Support at
    1-800-WiFi-LAN or support@arubanetworks.com
    [root@localhost mercury]#
    ----- 

     

    Attached are the release notes for AirWave 8.0.10.

    Attachment(s)



  • 2.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Dec 11, 2015 11:19 AM
    Thnaks Rob, finally have
    "AirWave now calculates the uptime of a device based on the actual uptime of the device, not just the time
    that the device is connected to the AMP."

    Would still love to see a time-stamp report of when a device went down and came back up.
    Feature request was added to the portal a while back ;)


  • 3.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Dec 17, 2015 02:39 PM

    I just performed an upgrade from 8.0.8.1 to 8.0.10.  According to the release notes it is bundled with CentOS 6.6, but I am still seeing OS version of 6.2.  I did perform the command that was specified at the end and rebooted.  Is the 6.6 only for new installations, or did I miss a step?



  • 4.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Dec 17, 2015 02:41 PM
    New installations afaik


  • 5.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Dec 23, 2015 09:03 PM

    I tried the process in the following article and was able to vaccum database and upgrade successfully.

     

    To vacuum/reindex please run the following commands on AMP CLI :

    1. Disable AMP:
    # amp_disable

    2. Prior to running each db command below (reindex and vacuum) stop and restart the postgres database using the commands:
    # service postgresql stop
    # service postgresql start
    This will remove any queued up nightly jobs and allow the reindex and vacuum jobs to proceed with without waiting in the queue.

    2a. If you are a super user for postgres then :
    #db
    airwave=> reindex database airwave
    airwave=> vacuum full analyze

    2b. For normal root login users, run the following commands:
    AMP Version < 7.3
    # /opt/airwave/bin/reindexdb -d airwave -Upostgres
    # /opt/airwave/bin/vacuumdb -v -f -z -Upostgres airwave
    AMP Version >= 7.3
    # /opt/airwave/pgsql/bin/reindexdb -d airwave -Upostgres
    # /opt/airwave/pgsql/bin/vacuumdb -v -f -z -Upostgres airwave

    2c. or you can also run these for normal root login users :

    # psql -Upostgres airwave
    psql (8.4.4)
    Type "help" for help.

    airwave=# reindex database airwave;
    REINDEX
    airwave=# vacuum full analyze;
    VACUUM
    airwave=#

    3. Enable AMP bye :
    # amp_enable

     

    Ref: http://www.airheads.eu/t5/Monitoring-Management-Location/Vacuum-Reindex-in-AMP/ta-p/177274

     



  • 6.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Dec 29, 2015 02:33 PM

    The linked article on vacuuming has some AMP version specific commands.

    Is that "safe" for general use?

    I'd feel nervous aobut following it unless AMP-Engineers had blessed it or extended it for the current version.

     

    I've got the "needs vacuuming" message from AMP_upgrade, so I'm currently awaiting a callback from TAC with instructions. (I'm also backing up my AMP database before anything happens)



  • 7.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Dec 29, 2015 02:56 PM

    The support team has a script that targets only the bloated tables.  Running vacuums on all tables is generally not needed.



  • 8.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 04, 2016 08:35 AM
    There is no way to revert so have to call in, in order for TAC to run the scripts correct?

    When I say revert, I don't mean, re-install with the previous version and restore backup..


  • 9.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 04, 2016 10:25 AM

    The database bloat check is pre-upgrade.  So your version shouldn't have changed if the check failed.



  • 10.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 04, 2016 10:37 AM
    Awesome.


  • 11.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 13, 2016 05:04 PM

    I'm having the same problem:

     

    "Database tables need vacuuming before you can proceed with upgrade"

     

    I ran through the reindix and vacuum procedure TWICE.  Still getting the same error when attempting to run the upgrade.  I'm at 8.0.9.2 right now.

     

    Anyone have any other ideas?  I will be getting in touch with Aruba support about it tomorrow, if I don't get it sorted out between now and then.  Thanks everyone :)

     

     

     



  • 12.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 14, 2016 01:36 PM

    There are some tables that even after vacuumed are still large enough to block the upgrade.  The script support has will pinpoint the specific tables that block the upgrade.  Support will either help vacuum the table further or patch to bypass the check based on the list of bloated tables.



  • 13.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 14, 2016 01:40 PM

    Yes, support gave me a hand with that earlier.  He just went into the upgrade script and changed the value of the bloat check limit to be slightly higher than our level of bloat.  But, I thought it was a big jump upward, and I couldn't understand the tech's explanation for why that was fine.  Why would the limit be set at aroung 1GB, and our biggest table was aroung 6GB, and so he just set the limit in the check to be 7GB?  If 6GB is perfectly fine, why would the engineers who created the check have used such a much smaller value for it?

     



  • 14.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 14, 2016 02:22 PM

    Great question.

     

    For some customers, the periodic vacuuming of the database was not completing due to the amount of db activity (the db vacuum process was falling behind).  This eventually caused some issues that the check is trying to pro-actively prevent.  Symptoms were large numbers of transactions waiting for db locks to write, sluggish UI, and poor db performance (lots of items pending in queue).  The initial thought was that it only happens for customers maintaining a long history, or had a lot of unique clients, but then it was observed on customers who didn't have the same scaling which prompted the decision for more aggressive vacuuming due to the lack of a distinct pattern.



  • 15.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 14, 2016 02:39 PM

    Ok, hmm..

     

    So, while I'm still scratching my head about database whoas..

     

    I've been seeing this message on my AMP for months:

     

    "Nightly maintenance failed because of unsuccessful jobs: clean_database. Contact Aruba Technical Support"

     

    And of course, I've been passive-aggressively ignoring it, daring it to manifest in some discernable problem going on somewhere.  But, nope.

     

    After successfully completing the upgrade to 8.0.10 (and additional kernel patches it also included) a couple of hours ago, I went ahead and, once again, ran through the reindex and vacuum procedures covered earlier in this thread.  I was hoping it would finally result in some shrinking of the size of the database(s).  But, it still doesn't appear to have.  I'm wondering if I'm going to see any difference in the results of the nightly maintenance.  I'm kind of expecting I won't, because that problem has remained outstanding through at least a couple of previous point release upgrades.  Is there anything else I can check or do proactively to try and fix, or make sure it's fixed already, this 'clean_database' routine that's been unhappy for so long?

     

    -John

     



  • 16.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 26, 2016 01:27 PM

    @jpjackso

     

    Are there any errors in:

    /var/log/nightly_maintenance

    /var/log/clean_database

     

    If there are, chances are that the clean database process isn't completing still.  I suggest opening a support case and providing the logs for support to help debug further.



  • 17.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 26, 2016 01:53 PM

    Actually, I have done that as part of the original case, and I'm still waiting for the engineer to get back to me now with next steps.  He said he needed to work with a higher level of support first, to try and figure out what the problem is.  That was a week or so ago, at this point.  What I'm hoping he'll come back with is instructions for manually doing a definitive cleanup of the database, even if we or I have to leave it for hours or days to complete, if need be.

     



  • 18.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 26, 2016 02:39 PM

    Understood.  If you happen to have a support case number, I can see about getting a status update.  And possibly help push toward a quicker resolution.



  • 19.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 26, 2016 02:42 PM

    That would be so nice of you!  :)

     

    Case #1816204

     



  • 20.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 26, 2016 04:31 PM

    Hm... I got word from support that the issue was resolved hours ago.  Are you still experiencing the clean_database message on the homepage?



  • 21.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 26, 2016 04:34 PM

    Yup, the message is still there, as usual.

     



  • 22.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 27, 2016 11:06 AM

    I pinged support escalations, so hopefully we'll can get that case closed for you soon.



  • 23.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 27, 2016 11:09 AM

    Rob thanks, that is much appreciated!  Are you saying you guys know the cause of that failure message, and what the fix is?  :)

     

    -John

     



  • 24.  RE: AirWave 8.0.10 now available. Upgrade has a database bloat check.

    Posted Jan 27, 2016 11:19 AM

    Not without seeing the log file.  I don't have access to see the actual support cases, but I've got some ideas for where to look and debug.  I can hide the error, but that's not the same as resolving the issue since it'll just come back again later on.