Network Management

yes i know the server is old but we are replacing it with a brand new server.

In the meantime, we had to perform an emergency reboot and now are seeing this error.

 

[root@aws1 scripts]# ./clean_database


Thu Jun  4 10:52:34 2015:
expiring old rows failed! ERROR:  index "client_historical_ip_client_historical_id_idx" contains unexpected zero page at block 2454
HINT:  Please REINDEX it.
CONTEXT:  SQL statement "DELETE FROM ONLY "public"."client_historical_ip" WHERE $1 OPERATOR(pg_catalog.=) "client_historical_id"": delete from client_historical where disconnect_time < extract(epoch from (to_timestamp(1433429554) - '7 days'::interval))  at /usr/local/airwave/lib/perl/Mercury/Utility/ChangeDB.pm line 838
        Mercury::Utility::ChangeDB::expire_old_rows('client_historical', 'disconnect_time', '7 days') called at /usr/local/airwave/lib/perl/Mercury/Utility/ExpireRows.pm line 112
        Mercury::Utility::ExpireRows::expire_client_historical('Mercury::Utility::ExpireRows') called at ./clean_database line 28
        main::__ANON__() called at /usr/local/airwave/lib/perl/Mercury/Utility.pm line 178
        eval {...} called at /usr/local/airwave/lib/perl/Mercury/Utility.pm line 178
        Mercury::Utility::safe_eval('CODE(0xaca9f0)') called at ./clean_database line 56
ERROR:  index "client_historical_ip_client_historical_id_idx" contains unexpected zero page at block 2454
HINT:  Please REINDEX it.
CONTEXT:  SQL statement "DELETE FROM ONLY "public"."client_historical_ip" WHERE $1 OPERATOR(pg_catalog.=) "client_historical_id"": delete from client_historical where disconnect_time < extract(epoch from (to_timestamp(1433429554) - '7 days'::interval))  at /usr/local/airwave/lib/perl/Mercury/Utility/ChangeDB.pm line 838
        Mercury::Utility::ChangeDB::expire_old_rows('client_historical', 'disconnect_time', '7 days') called at /usr/local/airwave/lib/perl/Mercury/Utility/ExpireRows.pm line 112
        Mercury::Utility::ExpireRows::expire_client_historical('Mercury::Utility::ExpireRows') called at ./clean_database line 28
        main::__ANON__() called at /usr/local/airwave/lib/perl/Mercury/Utility.pm line 178
        eval {...} called at /usr/local/airwave/lib/perl/Mercury/Utility.pm line 178
        Mercury::Utility::safe_eval('CODE(0xaca9f0)') called at ./clean_database line 56

Also, on the WebUI, clicking on an AP sometimes gives us this error page.

 

Please dont tell me I need to recompile (root; make)....i can't do that right now...:)

and because the ./clean_database script won't run, i cannot backup the server

Found this article
http://community.arubanetworks.com/t5/Monitoring-Management-Location/Vacuum-Reindex-in-AMP/ta-p/177274
not sure if it will help me...

The reindexing from that article should do the trick, but it has to be done while AMP is disabled or it will take a horribly long time.

Ya that's my problem. I need it online because of customer visibility.

It seems that the client_historical_ip table is foo bared and not accessible even with a select statement.

I am now worried that DOWNs are not being recorded properly. A new device discovery cannot even be added, I cannot un-ignore certain devices, etc.

This server sees anywhere from 9,000 to 20,000 clients a day so who knows how large that table is.....

The problem will probably prolong if the reindexing doesn't occur.  You might be able to open a support case and get some escalation assistance from a developer.  But any database manipulation should be done during a downtime/maintenance window.

Thanks Rob.

My DBA here says he can do the work but like you mentioned we should disable AMP for it go as fast as possible.

I have spoken to a TAC guy who has helped me in the past with AMP issues, waiting for his feedback...