Network Management

Hi,

I'm new to AW.  Installed it as a VM on my Dell PE T610 server, which has 4 other Windows VMs running.  I allocated 8GB RAM, but only 40GB disk space when I created my AW VM.  It has already consumed the entire 40GB!  Yet I only have 10 devices (1 AP) and 12 clients.  I can't fathom how so much space could get used up so quickly in the 60 days sent I installed AW.  Are there tricks to reduce disk usage?  Maybe I can change the default polling interval that SNMP devices get polled?  Are there any stats as to how much data a single poll of 1 SNMP device consumes?

Are there any system reports that might show how the AW server disk space is being consumed?

TIA,

Mike

Mike,

40GB is by far too low. I run a lab server with 80GB and it has 60GB in use. Realize that Airwave stores a lot of data, especially if you have AMON turned on and send AppRF data.

You can see the disk usage in System -> Performance -> (scroll down) -> Disk Space. Which does not tell where the data goes in, but it is likely to be in the database, in upgrade files, and in backups. You can clean up some of those via the AMPCLI Delete File option.

I would create a backup, spin up a new Airwave with more disk space, and restore the backup there.

Use the Airwave Sizing Calculator in ASE to get an estimate, which even sizes to 113GB disk for my lab size.

Thanks for the info.  I'm really surprised that the database gets so big for such a small amount of devices & clients.  I have Kaseya and Labtech servers accumulating data, including Windows event logs, for hundreds of computers and the databases are very small in comparison to AW.

I wonder if I could just edit the VM's disk size under ESXi4.1?  First I'd shut down the VM, then do the edit.  Will the Linux os see the added disk space?

Go to AMP Setup --> General --> Historical Data Retention.

Reduce the retention times down.  If I remember correctly reducing the 'client data retention interval' will have the biggest effect.  You should see this after the overnight maintenance, or you could change that to run in 10 mins time to get a more immediate cleanup.

If you go to System -> Performance, there's a widget on there (about 3/4 of the way down under Database Statistics) that shows the largest db tables.  Most likely the AppRF (PEF) and device config tables are usually among the heavy db space consuming tables.

Hmmm, maybe I don't have a disk space issue, as here is what AW's System->Performance->Disk Space shows about my 40GB.  I don't pretend to understand Linux (CentOS), but I'm guessing the graph tells me my disk usage is 45.9%?

Oddly, I still keep seeing this on the AW console, indicating out of memory errors.  What to do?  I have allocated 8GB RAM as per the minimum specs...

Looks like it's not your disk space that's full, it's the RAM that's being fully utilized.  OOMkiller comes into play when there's a process that's starved for memory.  Do you have a capture of the top portion of the System -> Performance page?  I'm interest in seeing how the RAM was setup on install.  You may have selected 8GB RAM, but it's not helpful if somehow the OS decided to put 4GB of it allocated to swap.

Also, since it's hitting OOMkiller - opening a support case to have someone take a closer look may help.  I'd start by looking at the low_level_service_watcher, service_watcher, kernel, and messages logs.

Below is the Performance info.  Indeed it does list 4GB swap space.  I don't recall ever making any choices about that.  How/where does one configure that?

It happens during the OS install when installing from the AMP ISO - you can get into the advanced disk settings when it does the disk selection.  The OVA install doesn't give any option.

I did install AW from an ISO for VMware - is that what you're calling "OVA"?

On the support site, there's 3 files.

*.tar.gz = upgrade package

*.iso = install iso

*.ova = install ova

If you use'd the ISO file, then you should have had an option at the OS install splash to enter 'manual' to allow for customizing the disk partitioning and RAM allocation.  If you ran 'install' it will default to a split calculated based on allocated disk space and RAM.  The default will usually take 2/3 for RAM, and then 1/3 into swap.

 I did install "AMP-8.2.5-x86_64.iso", but being totally new to AW I wouldn't have known about the manual setup options, nor would I know how to partition memory space.

So in my case, since I allocated 8GB RAM in VMware for this AW vm, for some reason it allocated a 50/50 memory/swap.  Is there any way to change it?   If not, I might try a fresh install and do the manual setup.  I'm a bit puzzled as to why the default settings wouldn't handle 10 devices (with 1 AP having 12 clients).  I do have to say, the GUI is slow as mollasses :(

But as I think back, I believe I initially only allocated 4 or 6GB RAM to the vm.  And initially I installed it in a datastore that resided on a NAS unit rather than my Dell PE server.  Once I saw that AW looked promising, I used the VMware vCenter Converted Standalone utility to copy the AW vm on to my Dell server's datastore.  I think I noticed that the copied AW vm no longer reported the os to be Linux - instead it simply says "Other (32-bit)".

Maybe I should delete this copied vm and start fresh?

Can I export the devices I've configured into AW so far, install a new AW vm, then import from that saved device list?

It usually depends on the VHost setup.

To save what you've done so far, grab a backup from System -> Backups.  If you've made changes since the last nightly maintenance time, you could force a backup from the AMPCLI menu - then SCP the file to a remote server (any linux server running sshd, or you can install bitvise sshd onto your win laptop, if on a Mac, you might have to update the ciphers to make scp work).

Then after fresh installing a new instance, upload the file - you have to initialize the SCP from AMP to the remote host to pull the file.  And then restore that backup file.

Workflow isn't great right now, but there's improvements already planned for 8.2.6 that will help alleviate some of the shortcomings.

This is interesting: I powered down the AW guest vm, then edited the VMware settings for the os - changing it to Linux, CentOS 4/5/6 (64-bit).  It was previously set at Other - unknown.  I then powered up the AW vm, but I'm unable to see any console - just black screen.  I figured that meant the os never booted.  Just for kicks I tried to log into the AW GUI - and got in!  It also seems a good bit faster on most of the pages.

The ISO's filename I installed suggests it's boith x86 and x64 - so how do I know what actually got installed?

Thanks for all your assistance!

If you hit System -> Performance, it should show you the kernel info.  Here's an example from one of my lab AMPs.  The whole kernel section is extracted into a pretty format from parsing the unix 'uname' cmd output.  x86_64 translates into 64-bit architecture.

The CLI should still be reachable though if the UI is reachable.  The CLI frame might be in sleep mode, you can try hitting space bar to see if that wakes the prompt back up.

OK, I was confused by the x86_64 - looked like it meant either x86 or 64 :)  So mine is 64 bit, and that's what the vm is configured for...

I guess AMP needed to rest overnight, as now the CLI is working.  But I still see the out of memory errors.  There's no way to partition memory and swap space differently post install, right?  Is there an English translation as to what those OOM messages mean?  It seems to be "killing" processes that always mention "httpd" - what are those processes?  And when I look at the rest of the OOM messages, it looks to be reporting the amounts of memory used by various things:

e.g. total-vm: 3,478,496kB,  anon-rss: 63,052kB,  file-res: 176kB

I'm guessing the first one is total memory used by vm (the AW guest vm under ESXi?).  That's showing about 3.5GB.  Add in the other two amounts and it's under 4GB.  And I guess I only have 4GB to play with, as the other 4GB is dedicated to swap space and does not count?

The httpd is apache process which serves up the UI and has children processes for other background processing of widgets.

It's possible some of the RAM might be lost due to VM overhead.  I prefer VMware 5.5 since I'm not a fan of the newer UI in 6.5.  I feel that things got a lot slower on my servers that were updated to 6.5 and I lose connectivity to those instances frequently.

I also find it somewhat surprising that the minimum sizing isn't working for you.  Though I've been typically allocating 4 CPU, 16 GB RAM, and 80 GB disk and never hit more than 5 devices being monitored (another team member handles the scale testing).  But my instances also don't live longer than a few months at most.

Yes, ESXi 4.1 has worked beautifully for me for years, it's free, so I haven't bothered to upgrade.  I know that all too often "new" features come at a price - bloated code...

Would be nice if Aruba came out with a "light" version of AW.  I'm probably only going to use a tiny fraction of it's capabilities, so a lighter version would probably be much faster as far as the GUI.  Hmmm - are there some services I could disable to lighten the memory load of AW server?

There are a few things that can be done to skip certain processing activities.  But there's really no on/off switch for some features.

What devices do you have setup?  Is it a controller & AP, or an Instant AP?  Any switches?

If it's Instant, then you can disable the AMON things from AMP Setup.  If you're not using VisualRF, that can be disabled from VRF Setup.  You can also change the time of nightly maintenance  (also on AMP Setup) so that it happens during off hours (otherwise it sometimes runs in the background if it didn't catch up during the maintenance cycle).

I currently have an IAP (AP-303H), and several Netgear switches (support SNMP), a Netgear router (also has SNMP).

I'll try disabling AMON and some other things and see if I can reduce the memory footprint.

If I stop the AW server, then edit the ESXi guest to have more memory (e.g. 12GB), when I restart AW will it recognize the addl memory?  Or is is set at install time based on what's available?  In the Windows world it would be dynamic, but I don't know much about Linux...

Expanding the memory requires an OS command to recognize the change.  With the new CLI, that action would require a support case - which is why I usually suggest just taking a backup and then doing the fresh install instead w/ restoring the backup after.