Monitoring, Management & Location Tracking

async_logger client is very large (>1 GB) and is filling up with "integer out of range" errors

Aruba Employee

Issue: async_logger client is very large (>1 GB) and is filling up with "integer out of range" errors

 

 

The async_logger_client log file is enormous (bigger than 1 GB). It rotated recently and the async_logger_client.1 file also is extremely large. The logs are filling up with error messages like this:

ERROR: integer out of range: insert into ap_discovery_event

(See more detailed error message below.)

Also the pgsql log is filling up with messages that look like this:

Oct 2 11:57:40 airwave postgres[12764]: [18835-1] ERROR: 22003: integer out of range
Oct 2 11:57:40 airwave postgres[12764]: [18835-2] LOCATION: int84, int8.c:1205
Oct 2 11:57:40 airwave postgres[12764]: [18835-3] STATEMENT: insert into ap_discovery_event (ap_id, discovering_ap_radio_index, discovery_time, last_discovering_ap_id, network_type, radio_mac, rssi, signal, type) values ($1, $2, $3, $4, $5, $6, $7, $8, $9)

 

WHAT'S GOING ON?

The source of the problem is with a database table that stores temporary discovery event data. Over the life of the AMP, the ID for records in that table continues to increase without being reset and can eventually exceed the maximum value allowed for an integer (approximately 2,000,000,000). This causes a flood of error messages that fill up the log files. You can confirm this by running the following command:

# dbc "select nextval('ap_discovery_event_id_seq');"


SOLUTION

The solution involves disabling the AMP temporarily, deleting all of the records from the ap_discovery_event table, resetting its ID counter, and re-enabling the AMP. To do this, run the following commands from the AMP command line:

# disable_amp
# dbc 'delete from ap_discovery_event;'
# dbc "select setval('ap_discovery_event_id_seq', 1);"
# enable_amp

 

This has been fixed in AirWave 7.1.


-----------------------------------------
More detailed error message:

ERROR: integer out of range: insert into ap_discovery_event (ap_id, discovering_ap_radio_index, discovery_time, last_discovering_ap_id, network_type, radio_mac, rssi, signal, type) values (?, ?, ?, ?, ?, ?, ?, ?, ?) (5944 1 1287589015 5965 3 00:22:33:CC:DD:60 -90 -90 1) at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 225
Mercury::DB::Q::execute('Mercury::DB::Q', 'insert into ap_discovery_event (ap_id, discovering_ap_radio_i...', 5944, 1, 1287589015, 5965, 3, '00:22:33:CC:DD:60', -90, ...) called at /usr/local/airwave/lib/perl/Mercury/DB/DiscoveryEvent/Action/CreateDiscoveryEvent.pm line 61
Mercury::DB::DiscoveryEvent::Action::CreateDiscoveryEvent::execute('Mercury::DB::DiscoveryEvent::Action::CreateDiscoveryEvent=HAS...') called at /usr/local/airwave/lib/perl/Mercury/DB/DiscoveryEvent/ActionSet/Base.pm line 26
Mercury::DB::DiscoveryEvent::ActionSet::Base::__ANON__() called at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 410
eval {...} called at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 397
Mercury::DB::Q::_run_then_do('Mercury::DB::Q', 'CODE(0x265abf9c)', 'CODE(0x265af164)') called at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 369
Mercury::DB::Q::run_in_transaction('Mercury::DB::Q', 'CODE(0x265abf9c)') called at /usr/local/airwave/lib/perl/Mercury/DB/DiscoveryEvent/ActionSet/Base.pm line 29
Mercury::DB::DiscoveryEvent::ActionSet::Base::execute('Mercury::DB::DiscoveryEvent::ActionSet::ExistingAP=HASH(0x265...') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Monitoring/DiscoveryEvent.pm line 51
eval {...} called at /usr/local/airwave/lib/perl/Mercury/Daemon/Monitoring/DiscoveryEvent.pm line 50
Mercury::Daemon::Monitoring::DiscoveryEvent::handle_event('Mercury::Daemon::Monitoring::DiscoveryEvent=HASH(0x25f49764)', 'HASH(0x25682e64)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Monitoring/RFNeighborSorter.pm line 60
Mercury::Daemon::Monitoring::RFNeighborSorter::process_events('Mercury::Daemon::Monitoring::RFNeighborSorter=HASH(0x242d23a4)', 'node_id', 'airespace_monitor.4017') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Monitoring/Controller/Rogue.pm line 63
Mercury::Daemon::Monitoring::Controller::Rogue::_handle_event('Mercury::Daemon::Monitoring::Controller::Rogue=HASH(0x2452999c)', 'Mercury::AP::Airespace::Airespace4400=HASH(0x2575394c)', 'HASH(0x24e98194)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Monitoring/EventHandlerBase.pm line 68
Mercury::Daemon::Monitoring::EventHandlerBase::do_handle_event('Mercury::Daemon::Monitoring::Controller::Rogue=HASH(0x2452999c)', 'HASH(0x24e98194)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Monitoring/EventHandlerBase.pm line 35
eval {...} called at /usr/local/airwave/lib/perl/Mercury/Daemon/Monitoring/EventHandlerBase.pm line 29
Mercury::Daemon::Monitoring::EventHandlerBase::handle_event('Mercury::Daemon::Monitoring::Controller::Rogue=HASH(0x2452999c)', 'HASH(0x24e98194)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/AsyncLoggerClient.pm line 98
Mercury::Daemon::AsyncLoggerClient::__ANON__('HASH(0x24e98194)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/AsyncLoggerClient.pm line 311
eval {...} called at /usr/local/airwave/lib/perl/Mercury/Daemon/AsyncLoggerClient.pm line 311
Mercury::Daemon::AsyncLoggerClient::main('Mercury::Daemon::AsyncLoggerClient=HASH(0x245183f4)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Base.pm line 154
Mercury::Daemon::Base::go('Mercury::Daemon::AsyncLoggerClient=HASH(0x245183f4)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Service/Forkable.pm line 53
Mercury::Daemon::Service::Forkable::__ANON__() called at /usr/local/airwave/lib/perl/Mercury/Utility/Processes.pm line 74
Mercury::Utility::Processes::__ANON__() called at /usr/local/airwave/lib/perl/Mercury/Utility/Processes.pm line 54

Mercury::Utility::Processes::fork_and_run('CODE(0x2451c744)') called at /usr/local/airwave/lib/perl/Mercury/Utility/Processes.pm line 76
Mercury::Utility::Processes::fork_and_run_detached('CODE(0x2452010c)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Service/Forkable.pm line 55
Mercury::Daemon::Service::Forkable::_restart('Mercury::Daemon::Service::AsyncLoggerClient=HASH(0x10b179bc)', 1) called at /usr/local/airwave/lib/perl/Mercury/Daemon/Service/MultiForkable.pm line 32
Mercury::Daemon::Service::MultiForkable::_restart('Mercury::Daemon::Service::AsyncLoggerClient=HASH(0x10b179bc)', 1) called at /usr/local/airwave/lib/perl/Mercury/Daemon/Service.pm line 87
Mercury::Daemon::Service::restart('Mercury::Daemon::Service::AsyncLoggerClient=HASH(0x10b179bc)', 1) called at /usr/local/airwave/lib/perl/Mercury/Daemon/ServiceWatcher.pm line 151
Mercury::Daemon::ServiceWatcher::check_services('Mercury::Daemon::ServiceWatcher=HASH(0x242a862c)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/ServiceWatcher.pm line 119
Mercury::Daemon::ServiceWatcher::main('Mercury::Daemon::ServiceWatcher=HASH(0x242a862c)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Base.pm line 154
Mercury::Daemon::Base::go('Mercury::Daemon::ServiceWatcher=HASH(0x242a862c)') called at /etc/httpd/conf/bootstrap.pl line 84
eval {...} called at /etc/httpd/conf/bootstrap.pl line 83
main::__ANON__() called at /usr/local/airwave/lib/perl/Mercury/Utility/Processes.pm line 54
Mercury::Utility::Processes::fork_and_run('CODE(0x2429d74c)') called at /etc/httpd/conf/bootstrap.pl line 93
require /etc/httpd/conf/bootstrap.pl called at (eval 2) line 1
eval 'require q/etc/httpd/conf/bootstrap.pl
;' called at /etc/httpd/conf/bootstrap.pl line 0
at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 225
Mercury::DB::Q::execute('Mercury::DB::Q', 'insert into ap_discovery_event (ap_id, discovering_ap_radio_i...', 5944, 1, 1287589015, 5965, 3, '00:21:56:CC:BC:60', -90, ...) called at /usr/local/airwave/lib/perl/Mercury/DB/DiscoveryEvent/Action/CreateDiscoveryEvent.pm line 61
Mer55
Mercury::Daemon::Service::Forkable::_restart('Mercury::Daemon::Service::AsyncLoggerClient=HASH(0x10b179bc)', 1) called at /usr/local/airwave/lib/perl/Mercury/Daemon/Service/MultiForkable.pm line 32
Mercury::Daemon::Service::MultiForkable::_restart('Mercury::Daemon::Service::AsyncLoggerClient=HASH(0x10b179bc)', 1) called at /usr/local/airwave/lib/perl/Mercury/Daemon/Service.pm line 87
Mercury::Daemon::Service::restart('Mercury::Daemon::Service::AsyncLoggerClient=HASH(0x10b179bc)', 1) called at /usr/local/airwave/lib/perl/Mercury/Daemon/ServiceWatcher.pm line 151
Mercury::Daemon::ServiceWatcher::check_services('Mercury::Daemon::ServiceWatcher=HASH(0x242a862c)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/ServiceWatcher.pm line 119
Mercury::Daemon::ServiceWatcher::main('Mercury::Daemon::ServiceWatcher=HASH(0x242a862c)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/Base.pm line 154
Mercury::Daemon::Base::go('Mercury::Daemon::ServiceWatcher=HASH(0x242a862c)') called at /etc/httpd/conf/bootstrap.pl line 84
eval {...} called at /etc/httpd/conf/bootstrap.pl line 83
main::__ANON__() called at /usr/local/airwave/lib/perl/Mercury/Utility/Processes.pm line 54
Mercury::Utility::Processes::fork_and_run('CODE(0x2429d74c)') called at /etc/httpd/conf/bootstrap.pl line 93
require /etc/httpd/conf/bootstrap.pl called at (eval 2) line 1
eval 'require q/etc/httpd/conf/bootstrap.pl
;' called at /etc/httpd/conf/bootstrap.pl line 0

 

while processing action set:

$VAR1 = bless( {
actions => [
bless( {
class => 'Mercury::DB::APDiscoveryEvent',
fk_id => '5944',
where => {
discovering_ap_radio_index => 1,
last_discovering_ap_id => 5965,
radio_mac => '00:21:56:CC:BC:60',
type => 1
}
}, 'Mercury::DB::DiscoveryEvent::Action::DeleteDiscoveryEvents' ),
bless( {
args => {
discovering_ap_radio_index => 1,
discovery_time => '1287589015.05148',
last_discovering_ap_id => 5965,
network_type => 3,
radio_mac => '00:22:33:CC:DD:60',
rssi => '-90',
signal => -90,
type => 1
},
class => 'Mercury::DB::APDiscoveryEvent',
fk_id => '5944'
}, 'Mercury::DB::DiscoveryEvent::Action::CreateDiscoveryEvent' )
],
fk_id => '5944',
raw_data => {
ap_folder_id => '49',
discovering_ap_radio_index => 1,
discovery_time => '1287589015.05148',
from_remote_ap => '0',
last_discovering_ap_id => 5965,
network_type => 3,
radio_mac => '00:21:56:CC:BC:60',
rssi => '-90',
signal => -90,
type => 1
}
}, 'Mercury::DB::DiscoveryEvent::ActionSet::ExistingAP' );

Version history
Revision #:
1 of 1
Last update:
‎06-27-2014 05:07 PM
 
Labels (1)
Contributors
Tags (2)
Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: 
Is this a frequent problem?

Request an official Aruba knowledge base article to be written by our experts.