Network Management

I have 97 devices in Airwave, which is running on a physical server.  The airwave sizing guide recommended a 75GB drive.  I started with a single 146 GB drive.  I upgrade from Airwave 7.3.1 to 7.4.0 about 2 weeks ago. Around the same time I enabled RTLS Collection.  My airwave server ran out of disk space last Thursday.  I opened a TAC case and got help freeing up about 12%, but it was suggested I put a second drive in.  I added a second 146 GB drive and had to build the server over again and restore from backup.  After adding the second drive, I started out last Friday with 55% free.  I lowered the historical data retention settings a good bit.  Airwave has been filling about 3% of the total disk space every day since then, and I will be out of space again in about a week.  I opened another TAC case and was told nothing was wrong, I would either have to lower the historical data settings even more or add MORE hard drive space.  I already have 3 times+ the recommended space so I disagree with that, but TAC isn't any help.  Anybody ran into this or know if something could be wrong?

This is an interesting issue. I am considering enabling RTLS as well, but it appears that the retention of this data is probably tied into the Client Data Retention Interval. If that is the case, I believe this should become an RFE (Request for Enhancement) to average out RTLS data after a certain interval. For example, once RTLS data is more than a day old, average it out to every 10 or 15 minutes. Otherwise keeping client data beyond a couple days is going to eat up storage.

That is abnormally high disk usage.  What historical settings are you changing?  Do you know what type of data is using all of the space?  Are you in an environment that has lots of unique users, like an airport or other public location?

I am having the same issue.

I found a file - visualrf_msg.log - that is over 147GB in size. Anyone know what that may be used for and how to stop it from getting so big?

Have you been doing a large amount of work within AWMS using VisualRF?

Also, try doing the following command:

tail -100 /var/log/visualrf_msg.log

This should help you to figure out what is flooding your visualrf log file.

Not doing a lot in VisualRF.

The tail command produces a lot of this:

2012-01-24 15:29:14,306 WARN  Message[2]   com.airwave.model.external.RTLSMessageTask invalid mac[null] used to lookup access point radios

As in about 200-300 such messages per second.

Any idea?

Chris, do you have RTLS enabled?  That's the feature that has APs send signal data directly to AMP so that VisualRF's device triangulation updated more aggressively.  That one message you pasted in looks like it's related to RTLS, and yeah, the amount of data we get could cause that log to have tons of data.

I do have RTLS enabled, although that is not something new. It has been enabled for well over 2 years.

I don't believe this log has been growing that long.

Any way to check the top of it?

Put in a support ticket, they had me stop RTLS, delete the log and they are filing a defect about RTLS.

Thanks for the help everyone.

Chris,

Saw your reply about TAC having you turn off RTLS. Just wanted to note I also saw your reply about being able to see the top of the log.

Try "cat /var/log/visualrf_msg.log | more" and then just cntrl+c after the first window pops up.

or you can do a "less /var/log/visualrf_msg.log" and type " : q " (thats a semicolon) to quit out.

Both those commands will let you read through the log file and will start at the top.

Thanks.

Has anyone received an update on this topic?  I just ran into the same issue.

Thanks,

What version of AMP are you running?  The original issue was fixed in 7.5.  If you're still seeing an issue, then opening a support case would be the next step.

7.4.8 - That explains why I did.  Thanks for the reply and time to upgrade....

I have one for you guys.

Airwave 7.5.5 with 1.5 TB of HD space in RAID-10 monitoring ~2000 APs and 28 controllers.

Running for 2 months now, 93% disk space usage. TAC is no help as they keep saying to reduce HDR. I have about 100Gb free right now.

No VisualRF, no RTLS. Airwave is monitoring APs in a public hotspot (largest coffee shop where I am from).

Have reduced Historical Data Retention as much as I can. I have to go in and delete the Airwave backups as they are over 5Gb each. I have deleted as many log files as I can but I have also noticed the following:

[root@aw-1 alternative]# ls -alh
total 5.2G
drwx------   2 root root 4.0K Nov  6 04:44 ./
dr-xr-xr-x. 25 root root 4.0K Sep 24 11:14 ../
-rw-r-----   1 root root  387 Nov  6 04:44 amp_license.txt
-rw-r--r--   1 root root 1.9M Aug 25 04:17 postgres_dump_11619
-rw-r--r--   1 root root 921M Oct 15 04:16 postgres_dump_15594
-rw-r--r--   1 root root 1.9M Aug 21 04:17 postgres_dump_16899
-rw-r--r--   1 root root 947M Oct 22 04:26 postgres_dump_20711
-rw-r--r--   1 root root 1.9M Aug 24 04:17 postgres_dump_20774
-rw-r--r--   1 root root 918M Oct 24 04:21 postgres_dump_23046
-rw-r--r--   1 root root 804M Nov  6 04:44 postgres_dump_23325
-rw-r--r--   1 root root 1.9M Aug 26 04:17 postgres_dump_2559
-rw-r--r--   1 root root 812M Nov  5 04:45 postgres_dump_28456
-rw-r--r--   1 root root 1.9M Aug 20 04:17 postgres_dump_29711
-rw-r--r--   1 root root 1.9M Aug 23 04:17 postgres_dump_30162
-rw-r--r--   1 root root 868M Oct 28 04:37 postgres_dump_3342
-rw-r--r--   1 root root 1.9M Aug 22 04:17 postgres_dump_7537

Attached is screenshot of HDR settings.

What am I to do??

We have 24x146Gb HD. The server is maxed out in terms of Hardware.

Is my only choice a SAN?

Here is the dilemma: I have to keep 1 year of "live" data on the server and 3 years historical off-site.

Any ideas?

The issue you're running into is most likely that you have many 1-time clients in your environment that create files that take up more disk space than necessary.

My first check would be the size of the RRDs (these are the flat files that take up the most disk space):

          # du -hs /var/airwave/rrd

And then the count of how many files takes up that space

         # cd /var/airwave/rrd; find . -type f | wc -l

This should give you a picture of how much space is being taken up by RRD per RRD.  But you can also drill down into one of the client directories to see the size of each RRD file per client.  Default size is 240K per flat file (this is with default HDR values).  Each client can will have at least 2 flat files, and each monitored device has at least 2 flat files per radio.

Your client data retention interval decides the size of these RRDs.  So currently, for a user who could possibly log in once in a whole year, you have RRDs of 365 days with several lines of 0s taking up space, while only 5-10 lines of the same RRD has actual data.  Note: changing this value will only take affect for new clients, past clients won't be adjusted until they have been aged out and then come back as a new client (this is a limitation of the RRDs).

The main reason that support is saying to keep changing the HDR settings is because that's the only way to find the balance that keeps the disk space somewhat clear for newer clients while maintaining the amount of data you require.  Airports, coffee shops, businesses that are on street level are all network types that would see similar behavior.

I think the requirement of maintaining 1 year of live data may not be practical unless you have a clear definition of what live data means to you.  Bending the definition, you could say all data in reports is live data.  With this definition, you could run reports regularly (daily, weekly, and monthly) and age out clients more aggressively.  This way, the reports maintain all the 'live' data you need, while you free up the rest of your disk space for the roaming clients, and you can maintain these reports off server in a separate email account meant to preserve these reports, or PDF export these reports (with AMP 7.6).

An example of this would be to set HDR:

          Inactive clients = 30 days

          Client Association = 7 days

          Reports = 365 days

          Client Data Retention Interval = 30 days

This will define it so that clients who haven't logged into your network in a week will be considered Inactive.  Inactive clients who haven't returned in a month will be aged out (may want to be more aggressive here, but this is for the case of monthly reports).  Reports can be maintained for a full year (to satisfy the year of live data).  And then Client Data Retention gives you at least 30 days of reportable data for that user.

This can be further explored with support as they've seen multiple customers do the same and can advise on what the other customers are currently doing.

Hi Rob,

As always your responses are very detailed and awesome.

Basically my RRD directory is 1.3 TB.

[root@aw-1 mercury]# du -hs /var/airwave/rrd
1.3T    /var/airwave/rrd
[root@aw-1 mercury]# cd /var/airwave/rrd; find . -type f | wc -l
11892490

I have seen variations between 240Kb and 206Kb.

I believe a definition of what data is to be kept is in order.
With reporting alot of the times, the report is too big to be shown in detail in an email and it asks you to log into Airwave to see the rest of it. WIth adjusting HDR and reporting, how will this be affected?

Glad to see 7.6 has PDF functionality.

EDIT: What would you recommend as a client data report that should be kept, which values, do you recommend a specific custom report with which values

I neglected to mention it, but in addition to being able to export reports in PDF format, you'll also be able to have AMP generate the report and send it as a PDF attachment instead of an html formatted report.  This can resolve some of the large report issues, but here's a few other tips:

1.  If emailed reports grow to be too large, change the sample size.  A daily report is going to be shorter than a weekly report.  If you want to get even more granular, you can break it up even further by having reports by the hour, or every other hour.  Though even when I do have smaller reports, I still like to have a good summary report, meaning I'd still run a daily report, but with less output than the hourly breakdowns.

-- For a coffee shop, I'd say that probably an hourly report across each of the peak hours, and then a combined report for slower hours.  You may end up with 24 reports for a day if you have a constant peak of clients coming and going.  Or you may have as few as 12 reports if you break it down into a report for every 2 hours, etc.

2.  Another thing to try is to cut down the data displayed in the report.  I typically recommend running a report with all options selected, that way you get a good idea of what data the report yields.  From there, pick and choose the main portions that you'd want to see.

3.  If you're purpose is to track down customer usage, then you may have a single daily report for your workers who authenticate to a hidden internal SSID.  And then run separate reports for the clients based on their SSID.

------

For testing, I have 5 reports a day.  4 reports cover 6 hour periods.  The last report is a full daily report.  These reports currently arrive in my email box, which I can search through since they are HTML formatted.  If you go the PDF route, you may need to device a script of some sort that can look into each PDF file to parse for a specific MAC address.

An example of a trending report I have includes:

Session Data by OS, Session Data by Connection Mode, Session Data by SSID, Session Data by Role, Session Data by VLAN, Top Clients by Total MB Used by folder (showing top 20 users), the Client Session Summary, and then details for every Sessions and Sessions by Client.

An example of a client session report I have for just RAW DATA:

Details for every client and session.  None of the other bells and whistles.

These both email to me at my email address, but then I have email filter that sort by Subject.  The Trend emails go to a single folder, and then the RAW DATA goes to another folder.

Note -> You do not want to use the 'rerun' report option if you age out your data more aggressively.  Rerun only works if you maintain the data in your database, but we're trying to steer you away from that to save your disk space.

It really depends on how you'd like to search through the data that would govern how you configure the reports.  For my purposes, my reports show correlation between when the most users are on my network, and which device type they are typically using.

Great. I will be meeting with some executives today to discuss the service offering and what we should include in the report.

I will design some reports as a baseline and tweak as I go.

I will come back with some answers later today in terms client data.

As a side note, I am seeing the following in /var/airwave-backup

-rw-r-----   1 root apache 3.8G Nov  7 08:23 nightly_data001.tar.gz

-rw-r--r--   1 root root   656M Nov  7 08:23 nightly_data.tar.gz

2 backups for the same day? nightly_data001 is the correct backup right?

This Airwave solution is part of a Airwave failover as well and what I am showing you here is the primary appliance.

You're running AMP 7.5.6?

You may want to check and see if nightly maintenance is still running:

# psg night

If you see that nightly maintenance is running, check to see if the tarball is still being built:

# psg tar

During nightly maintenance, the postgres_dump file is created in /alternative, and then nightly_backup.tar.gz is built.  When the tarball is complete, then the rotation script proceeds.  Seeing nightly_backup.tar.gz with no numbers leads me to think that nightly maintenance is still running.

7.5.6 it is

here is the output

[root@aw-1 mercury]# psg tar
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     16628  0.0  0.0   9184  1256 ?        S    04:38   0:00 /bin/sh /usr/local/airwave/bin/stopped_amp_data_backup -o /var/airwave-backup/nightly_data.tar.gz
root     17005  3.6  0.0  19652  1684 ?        D    04:46  15:27 tar -hcvzf /var/airwave-backup/nightly_data.tar.gz /var/log /tftpboot /var/airwave /alternative/postgres_dump_16628 /alternative/amp_license.txt /etc/ntp.conf /etc/postfix/main.cf
root     23325  0.0  0.0   9184  1252 ?        S    Nov06   0:00 /bin/sh /usr/local/airwave/bin/stopped_amp_data_backup -o /var/airwave-backup/nightly_data.tar.gz
root     23690  3.6  0.0  19652  1640 ?        D    Nov06  68:51 tar -hcvzf /var/airwave-backup/nightly_data.tar.gz /var/log /tftpboot /var/airwave /alternative/postgres_dump_23325 /alternative/amp_license.txt /etc/ntp.conf /etc/postfix/main.cf
[root@aw-1 mercury]#

That's what I was expecting.  For some reason, creating the tarball takes a long time.

If you check the time stamps of the other backup files in /var/airwave-backup, what's the last modified time?  I'm thinking that most backups will end near that time range.  The good thing is that it's a background operation, and shouldn't affect the data collection.

Would love to tell you but I deleted them because I needed the space as I am running out of space :)

By now that backup should have completed.  Take a look at the last modified time of the backup before you pull it off server.  And then check to see if tomorrow's backup completes around the same time.

This is what I got

[root@aw-1 airwave-backup]# ls -lah
total 8.8G
drwxr-x---   3 root apache 4.0K Nov  7 13:28 ./
drwxr-xr-x. 22 root root   4.0K Aug 20 00:06 ../
-rw-r-----   1 root apache 3.8G Nov  8 08:38 nightly_data001.tar.gz
-rw-r-----   1 root apache 5.1G Nov  7 13:28 nightly_data002.tar.gz
drwxr-x---   2 root apache 4.0K Aug 20 00:00 watched_amps/
[root@aw-1 airwave-backup]# psg night
root     16624  0.0  0.0   9184   572 ?        S    Nov07   0:00 /bin/sh /usr/local/airwave/bin/nightly_maintenance
root     16626  0.0  0.0   9180  1216 ?        S    Nov07   0:00 /bin/sh /usr/local/airwave/bin/nightly_backup
root     16628  0.0  0.0   9184  1248 ?        S    Nov07   0:00 /bin/sh /usr/local/airwave/bin/stopped_amp_data_backup -o /var/airwave-backup/nightly_data.tar.gz
root     17005  3.9  0.0  19652  1676 ?        D    Nov07  65:34 tar -hcvzf /var/airwave-backup/nightly_data.tar.gz /var/log /tftpboot /var/airwave /alternative/postgres_dump_16628 /alternative/amp_license.txt /etc/ntp.conf /etc/postfix/main.cf
[root@aw-1 airwave-backup]# psg tar
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     16628  0.0  0.0   9184  1248 ?        S    Nov07   0:00 /bin/sh /usr/local/airwave/bin/stopped_amp_data_backup -o /var/airwave-backup/nightly_data.tar.gz
root     17005  3.9  0.0  19652  1676 ?        D    Nov07  65:34 tar -hcvzf /var/airwave-backup/nightly_data.tar.gz /var/log /tftpboot /var/airwave /alternative/postgres_dump_16628 /alternative/amp_license.txt /etc/ntp.conf /etc/postfix/main.cf

 Looks like it finished at 13:28? Nightly maintenance starts at 315am

This is a known limitation to using tar to create backups.  It wouldn't get much faster on a SAN since this is a software limitation, not hardware.  The main issue is the number of files that are being compressed and then packaged in the tarball (historical clean up and age out happens prior to the building of the backup tarball).  This can be mitigated with more aggressive age out, but that depends on if the reports will be enough to constitute live data.  In the meantime, it's worth it to keep a short log of backup completion times so that you have a relative ballpark for when this process completes.  AMP upgrades should be scheduled for when this backup process is not running.  Meanwhile, I'll bring it up in the next product discussion.

The backups usually finish between 1pm and 2pm.

I have reduced HDR once more and will be designing some reports based on your suggestions in a previous post.

I am just afraid that with the addition of the reports I will lose even more HD space as I do not have more room to play with.

No buffer.

I am at 96% full 60Gb left

I come in this morning to see this

[root@aw-1 mercury]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      1.5T  948G  449G  68% /
tmpfs                  32G     0   32G   0% /dev/shm
/dev/sda1              97M   36M   57M  39% /boot

 I am very happy but also thinking what the hell happened??

We reduced the HDR some more, more importantly, the client data from 365 to 185 days.

Could this have been the cause?

Yes, the changes made to the HDR is going to be directly linked with the amount of disk space being used in your setup.

Easiest way to verify is to double check the count of the RRDs and the size of RRD files (compared to previous numbers you gave earlier in the thread):

# cd /var/airwave/rrd; find . -type f | wc -l

# cd /var/airwave/rrd; ls -lahR | less

-- When doing this command, you can use ctrl+f to page forward (essentially VI navigation commands) and see relative RRD sizes.  You're going to have a mix of the older sized RRDs plus some smaller RRDs.

Nightly_maintenance most likely aged out a lot of your RRDs post HDR change, and then the newer RRDs generated are going to be smaller file sizes.  This is opening up more space on your disk (which was part of my goal in my earlier suggestions for changing the HDR values).

Ya exactly.

I am now in the process of designing reports in order to age out more aggresively based on the suggestions you gave earlier.

Again thank you.

I kind of wish this sort of information would be published somewhere like a sort of tips and tricks.

More on this.

I found out TAC had turned off AMON while they were tweaking my AMP at the beginning.

I turned it back on and I believe that I will run out of diskspace again even with the HDR settings in place from previous.

70% use, time will tell.

Keeping an eye on it.

But it looks like that nightly maintenance is constantly running on the primary server. >5gb backups so may take forever to TAR

Tarring does take a long time.  This is due to the sheer number of rrds that are being tarred up.  What's the current sizing of your AMP server (CPU / RAM / HD)?  Also, what's the load on the server in terms of devices, clients, number of vrf floor plans?

CPU
Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz Hyper-Threaded 8 Cores 20480 KB cache (2399.920 MHz actual)
Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz Hyper-Threaded 8 Cores 20480 KB cache (2399.920 MHz actual)

Memory
Installed Physical RAM: 62.87 GB
Configured Swap Space: 4.00 GB

Yep, with that amount of clients, it's going to be just straight RRD work being done by the backup process that's taking so long.  The best way to shorten that is to keep a shorter Historical Data Retention value.  The alternative would be to consider putting a SAN in the background.

Thanks for the info. HDR has been reduced extensively.

We should have setup a SAN from the beginning.

I'll live with the 1.5 hour failover :)

That's what happens I guess when you AMP is monitoring a Hotspot with so many clients :P

That log file certainly shouldn't be that big. I think it would be wise to open up a support case. 

If your hard drive is in danger of filling up, you could save off the last 1000 lines, then safely empty the original file's contents like this:

# tail -n1000 /var/log/visualrf_msg.log > /tmp/visualrf_msg.log   

# echo > /var/log/visualrf_msg.log

Agree with Dan.

Although shouldn't the log rotation be doing this automatically? If his visualrf log is filling up that much in 24 hours, I think there is a bigger concern at hand.