Network Management

We have a AW-2500 8.0.6.3 install monitoring just under 1,900 APs. We're running it on a nice big bare metal server with 2 x 8core E5-2660 CPUs, 94GB ram and an array of about 16 disks which (I believe) are 15k SAS.

 

On visual RF we currently have 22 buildings, most are three floors, with 255 APs added so far.

 

As far as I can see we're well within the hardware sizing guide for our AP deployment. However AMP is struggling. It's using all the available ram most of the time and then hitting swap so there are times the system starts to thrash.

 

Essentially it's unusable at times, as it starts thrashing. As you'd expect the problem is ram/disk bound as the CPUs never break a sweat.

 

Before I raise this with TAC does anyone have any tips for obviously daft things we may have done with the config?

I'm not an Airwave expert, but curious to know what service(s) are taking up all your RAM.  You can see this under System > Performance.

Dipping into swap used to be a panic point, but several performance enhancements have happened since.  What you may be seeing is light caching that's not yet reclaimed by the system.

 

In addition to the already suggested system performance values, how much RAM is allocated specifically to VisualRF (see VisualRF -> Setup -> Memory Allocation)?  This is RAM set aside for VisualRF use only.

 

Also, what are the numbers of threads allocated to data processing?  This is on AMP Setup -> General -> Perfomance box.

 

Do you have any large memory consuming features in use?  AppRF, UCC, IGC (instant gui config), large report runs (monthly/yearly - if you do run such reports, it's best to run them at times when the network is not at peak)

 

Another thing to check is the timing for when nightly backups complete:

# ls -la /var/airwave-backup

Ideally you want the backups to complete prior to peak network hours.  You can adjust timing of backups in AMP Setup -> General.

 

Can you also expand more on the sluggishness?  Is it pages with graphs that are loading slow?  Pages with list tables?  Rule of thumb is does it take longer than 10-15 seconds for these items to load.

Hi Rob, thanks for your input again :)

To answer each of your questions....

I'm seeing swap and all ram being maxed out for extended periods so I don't think it's caching.

VisualRF currently has 10GB allocated.
Monitoring processes - 24
Config processes - 40
Audit processes - 40
SNMP fetcher - 4

AppRF is turned on, I can check if this is something we feel we need at this time.
UCC is on, igc im not sure.
I don't think we have any report running going on.

When things get sluggish we see slow response of a previously fast ui. When it's really bad the server returns a blank white page and the entire ui is inaccessible. So we do see total loss of the GUI.

Just curious, but are you seeing the same behaviour across different browsers?

 

Have you rebooted the airwave server?

I haven't tested other browsers, I'm using Chrome. Can try Mozilla. The server has been rebooted recently after the 8.0.6.3 update and some centos updates, including a recommended kernel patch.

We've taken a look at our server this morning. What we've noticed is a ramp up of ram use in January. This would fit with upgrading our controllers from 6.3 to 6.4.

 

In January as our users returned (we're a university campus) we saw ram go up very quickly from the peaks of 50GB through December to maxing out everything available.

 

Today we've disabled UCC and AppRF to see what happens. Currently it's looking OK, having forced restart httpd RAM use is at 20GB and slowly climbing. I'll give it a couple of hours and see what happens.

 

We're not routinely doing any UCC comms across our wireless network, and for most users the firewall doesn't currently allow this. 

 

Current thoughts are a presumption that the switch to 6.4 on the controllers has thrown a lot more data at airwave. Of course this could still be a bug as we wouldn't expect everything to completely max out, especially as we appear to be within the sizing guide.

Did you have amon enabled on 6.3? 


Thanks, 
Tim

Hi Tim,

 

Yes, I'm told that we did...

 

Also I've clarified somethings. Apparently we went to 6.4 on the controllers in early December. Airwave was updated in late December and again last week.

 

So far the only change we can think of that was made between healthy Ram usage in December and the problems starting in January was the adding of more VisualRF floor plans. But we don't have all that many to be honest, and VisualRF has 10GB assigned to it so it should be using more than 40GB.

 

Just checked the server and our RAM use is still steadily climbing upwards, so we're passed what was being used in December.

This is about the point where opening a support case would be good.  It's worth it to investigate and see if there's a memory leak or bloat going on.

 

Prior to opening the support case, is there a specific time of the day that the slowness occurs, or is it like a tipping point where after X hours the system struggles?

 

I also suggest double checking some of the logs for errors, here's a few good ones to double check:

/var/log/httpd/error_log

/var/log/pound

/var/log/service_watcher

/var/log/httpd_performance

/var/log/async_logger_client

 

Also, try enabling:

# qlog enable rates

(this logging will continue until disabled and feeds into /var/log/amp_diag/rates)

Once the case is resolved or to make sure the hard disk doesn't get filled up (beyond log rotation, since this log catches a lot):

# qlog disable all

Here's what occasionally appears in error_log

 

[Mon Feb 09 09:13:38 2015] [error] [client 127.0.0.1] encountered object 'Apache2::RequestIO::print: (103) Software caused connection abort at /usr/local/airwave/lib/perl/Mercury/Apache/Request.pm line 109', but neither allow_blessed, convert_blessed nor allow_tags settings are enabled (or TO_JSON/FREEZE method missing) at /usr/local/airwave/lib/perl/Mercury/Utility/JSON.pm line 29, <GEN6> line 4.
at /usr/local/airwave/lib/perl/Mercury/Utility/JSON.pm line 29
\tMercury::Utility::JSON::encode_json('APR::Error=HASH(0x7fb5cfe2e360)') called at /usr/local/airwave/lib/perl/Mercury/Handler/Mocha/Base.pm line 75
\tMercury::Handler::Mocha::Base::print_error('Mercury::Handler::Mocha::ListView=HASH(0x7fb5cfd0c428)', 'Mercury::Apache::Request=HASH(0x7fb5cfd2b0c0)', 'APR::Error=HASH(0x7fb5cfe2e360)') called at /usr/local/airwave/lib/perl/Mercury/Handler/Mocha/Base.pm line 62
\tMercury::Handler::Mocha::Base::build_output('Mercury::Handler::Mocha::ListView=HASH(0x7fb5cfd0c428)', 'Mercury::Apache::Request=HASH(0x7fb5cfd2b0c0)') called at /usr/local/airwave/lib/perl/Mercury/Handler/Dispatcher.pm line 131
\tMercury::Handler::Dispatcher::__ANON__() called at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 663
\teval {...} called at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 650
\tMercury::DB::Q::_run_then_do('Mercury::DB::Q', 'CODE(0x7fb5cfe13470)', 'CODE(0x7fb5cfd0c638)') called at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 621
\tMercury::DB::Q::run_in_transaction('Mercury::DB::Q', 'CODE(0x7fb5cfe13470)') called at /usr/local/airwave/lib/perl/Mercury/Handler/Dispatcher.pm line 134
\tMercury::Handler::Dispatcher::call_handler('Mercury::Handler::Dispatcher', 'Mercury::Handler::Mocha::ListView=HASH(0x7fb5cfd0c428)', 'Mercury::Apache::Request=HASH(0x7fb5cfd2b0c0)') called at /usr/local/airwave/lib/perl/Mercury/Handler/Dispatcher.pm line 56
\tMercury::Handler::Dispatcher::handler('Mercury::Handler::Dispatcher', 'Apache2::RequestRec=SCALAR(0x7fb5cfc24c40)') called at -e line 0
\teval {...} called at -e line 0
at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 670
\tMercury::DB::Q::_run_then_do('Mercury::DB::Q', 'CODE(0x7fb5cfe13470)', 'CODE(0x7fb5cfd0c638)') called at /usr/local/airwave/lib/perl/Mercury/DB/Q.pm line 621
\tMercury::DB::Q::run_in_transaction('Mercury::DB::Q', 'CODE(0x7fb5cfe13470)') called at /usr/local/airwave/lib/perl/Mercury/Handler/Dispatcher.pm line 134
\tMercury::Handler::Dispatcher::call_handler('Mercury::Handler::Dispatcher', 'Mercury::Handler::Mocha::ListView=HASH(0x7fb5cfd0c428)', 'Mercury::Apache::Request=HASH(0x7fb5cfd2b0c0)') called at /usr/local/airwave/lib/perl/Mercury/Handler/Dispatcher.pm line 56
\tMercury::Handler::Dispatcher::handler('Mercury::Handler::Dispatcher', 'Apache2::RequestRec=SCALAR(0x7fb5cfc24c40)') called at -e line 0
\teval {...} called at -e line 0
, referer: https://<servername>/ap_list

 

 

/var/log/pound all looks ok

 

/var/log/service_watcher contains many:

Mon Feb 9 15:35:13 2015: Starting additional Client Monitor Workers

 

/var/log/httpd_performance looks ok

 

/var/log/async_logger_client

This is a very large file... currently 3.8GB async_logger_client.1 was rotated this morning at 6.9GB

 

What time is set for nightly backup on AMP Setup -> General?

What's output of:

# ls -la /var/airwave-backup

 

The error output and size of log prior to rotation makes me think that something in the backend is having issues.  What's the difference in days for the async logger logs?

# ls -la /var/log/asyn*

 

Can you open a case with support and make sure they get a copy of the async logger client logs?

Hi Rob, nightly maintenance is at 04:15

 

The async logs are rotating within a day. There's only two of them, both dated today.

 

I'll open a support case and see where we get to. Thanks for your input.

Any update? I have the same problems but I'm trying resovler, exhausting all before reaching the TAC

RafaP,

 

Just call TAC.  That's what they are there for.

So system doesn't have anything that shows what processes of Airwave are using ram, there's something that show kernel, amp, buffers... It's all amp.

Interestingly we didn't see this problem until January. There's been some updates to Airwave but the main change is we've added more floor plans to visualrf. However this ram usage doesn't seem at all right to me:

Had similar performance issues with 8.0.6.3 I believe. Currently running 8.0.8 and the performance is much improved. Willing to try an update to correct the issue?

I used the 8.0.7.1 version and yesterday updated to 8.0.9.1 also used the best pratices of version 8.0 which mattered. Yesterday here the behavior is normal and will observe anything will do as Joseph recommended. I appreciate the comments