Network Management

With the upgrades to ArubaOS-Switch monitoring and management in AirWave 8.2.7, it seems like AMP Device Events are fetched directly from the main event log instead of being retained per device. When the main event log apparently has a hard limit of 1000 lines, which fills up pretty quickly with 240 switches, 780 access points and "Restarting service Client Monitor Worker" popping up every couple of minutes, I have no way of knowing what happened to a downed switch on friday when I get back to work on monday. If something serious happens, which is when I really need to examine the logs, we're talking hours. Access points, Cisco switches and Aruba MAS still retain the last 30 events, regardless of age.

Is there any way to store events for HPE Aruba switches per device, or at least massively increase the main event log limit?

That 1000 line limit is only applicable for UI display.  If you generate the diagnostics tarball from System -> Status page (link is near top of page), you can extract and view the full AMP events log (path var.log.system.amp_events).  The actual log observes a sizing rotation - some of my lab AMPs haven't hit that mark to rollover yet, with a first entry timestamp from 2 yrs ago, and about 10k lines of entries.

Undoubtedly promising, if the diagnostics tarball hadn't failed to generate. System -> Download Log Files -> amp_events, on the other hand, worked just fine, and yielded a log of 3736 lines starting october 27th. That is an artificial limit, as the event log was somehow limited to 10 hours instead of the preset 10 days just after the update. I fixed that, so I'm hoping the actual limit will do the trick for me.

That solves the problem with missing data, now I just need it to show up on each device. While the 1000 line UI limit is fine for the main event log, it's less than ideal to manually download and search through the full event log whenever I need to see what happened to a device a couple of days ago, and impossible for support staff with limited access.

Is that something TAC might be able to fix, or will it need to be fixed in an upcoming update?

I see what you're getting at, great idea.  It'd be a feature request to enable log visibility to specific roles.

Are you thinking about downloading the complete event log for non-admins? That would undoubtedly be useful, but doesn't address the issue of events being hidden in device view.

I think there's merit in that.  I haven't seen many feature requests about logging visibility.  But adding more logs to that download log section, and adding a role option for log visibility are feature requests that haven't been filed yet in the innovation portal.

There's definitely merit to that. I'd love to be able to customize role access, like giving our support staff access to more logs and the troubleshooting tab in the new switch management pages, but I'm guessing that's a bigger, and relatively unrelated, task. What I need now is easy access to my missing device events.

I'm also generally worried about only reading from the central event log. Yesterday one of our locations went down a few times due to fiber issues, and its 5 switches and 30 access points quickly generated 1500 log lines, of which 1300 were AP related. If that were to happen with our biggest location, with 46 switches and 179 access points, the 10000 line log would fill in an instant. Just running a firmware upgrade would get pretty close. If it's not an option to go back to storing events per device, something like a separate log for wired devices would go a long way to mitigate that.

There should be parsing to display the events per device.  On the Device's monitoring page, there's the option to click to 'Alerts & Events' - that page has a section for Device Events, and if more details is needed, there's the Audit log link at the bottom.  The Audit log is a separate link since it displays more history than the Device Event log.

That's not entirely accurate. While the aptly named audit log does go way further back than the event log (so it's apparently still possible), it merely tracks configuration changes, and has nothing to do with device events.

Today I experienced how useless the AMP Device Event log has become. I went to investigate a major event that happened sometime tonight but lo and behold, there was next to no information available. The event had apparently caused the event log to max out, and instead of rolling over, it decided to start fresh. So now I'm left with a log of 287 useless lines.

To add insult to injury, this occured at 04:02 AM, 13 minutes before the nightly backup, so whichever awkward workaround I try, I'll have no clue as to what happened tonight. That didn't go over well with my boss...

Seriously, guys - what gives?

Is there a support case opened?  Even if the log rotates, there's no ceiling cap on the log file since it doesn't blow away.  The previous log after rotate should be in a pigz tarball.

# ls -sh /var/log/system/*event*
1.5M amp_events
108K amp_events.1.gz
76K amp_events.2.gz
120K amp_events.3.gz
108K amp_events.4.gz

If you're in the CLI:

# unpigz amp_events.1.gz

to expand the previous day's log

There should be no circumstance where the log files would mysteriously disappear.

Phew... Good to know that was a bug, not a feature :-) I'll see about contacting TAC to try to find out why and run those log commands for me.

Back to the device event history - have you noted that as a feature request, or do you want me to formally request it somewhere?

Update: I got diagnostics tarball to generate via cli, and it contains complete event, audit, config and command logs for every single device. Seeing as each device page already uses the per-device audit log, it seems logical that it should do the same for events. At least the information is readily available, unhindered by any limits on the central event log.

it's best if you can submit the request.  innovate.arubanetworks.com is the portal address.  if you can't access, let me know and i can try - it just looks better when PLM sees that there's an actual customer attached.

Thanks, I'll do that :-)

Feature request: https://innovate.arubanetworks.com/ideas/NMS-I-1116