Wireless Access

Reply
Super Contributor I
Posts: 274
Registered: ‎04-04-2014

Tips for tweaking debug logging?

 

We have a hard-to-diagnose problem we are working with the TAC.  We have to debug a problem that occurs only a couple of times a day, and which we cannot depend on external syslog to reliably deliver debug messages for, since it may involve switching drops on the control plane.  We are trying to capture at the debug level of logging during the problem on the controller, however that level of logging easily wraps the 75Kish system log buffer by the time we can get in and show the logs, much less tar them up.

 

We are currently trying to guess which of the most noisy debug level processes we can turn off and then individually turn debug on for as many other processes as we can, each by hand, since it does not appear one can override a "debug" with a more specific "info" or "errors" setting.  However, this will probably not be sufficient to prevent a log wrap on the services we are interested in.

 

So two specific questions:

 

1) Is there any way to allocate larger (potentially very large) logging buffers on the controller?  Would that involve a reboot?

 

2) Could someone explain how to use "logging level X ap-debug" since we cannot seem to find documentation for the required "name"

parameter (online help says "Debug String Range 1-63")?

 

Other than that, does anyone have any other tips or tricks that might be useful in this situation?

 

Guru Elite
Posts: 21,517
Registered: ‎03-29-2007

Re: Tips for tweaking debug logging?

bjulin,

 

To get the specific debug message required, TAC must provide that information.  If it involves "drops" in the control plane, they need to replicate that in their lab, because they have specialized methods of obtaining that information that is not user-facing.



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Super Contributor I
Posts: 274
Registered: ‎04-04-2014

Re: Tips for tweaking debug logging?

 

Unfortunately TAC has not been able to replicate the problem, so I am trying my best to get whatever information I can out of our production setup.  Any help would be appreciated.

 

Guru Elite
Posts: 21,517
Registered: ‎03-29-2007

Re: Tips for tweaking debug logging?

bjulin,

 

If you feel that you are getting nowhere with TAC, you should escalate it.  You have not given us enough detail in your post to determine what is the issue and as a result, we cannot give you enough information to resolve it.  Please escalate with TAC.

 



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Guru Elite
Posts: 8,765
Registered: ‎09-08-2010

Re: Tips for tweaking debug logging?

The only thing I can think of based on the information you posted would be to write a script that copies the logs off to a tftp or scp server every X minutes.


Tim Cappalli | Aruba Security TME
@timcappalli | timcappalli.me | ACMX #367 / ACCX #480
Super Contributor I
Posts: 274
Registered: ‎04-04-2014

Re: Tips for tweaking debug logging?

 

cjoseph -- To be clear, I'm not asking for help with this particular problem.  I'm already "escalated" according to TAC and have also asked my SE to advise on the case.

 

My questions, above, are intended to both help me with this case and with future cases, and help anyone who happens across this thread.  So are there answers?

 

cappalli -- Thanks for the suggestion.  I've manage to tame the logs enough it just might be feasible to do so; we'll see once the issue occurs again.  Hopefully I haven't excluded essential messages.

 

Anyone know what setting controls 'nanny' logs that get enabled with "logging level debug system?"

 

 

Moderator
Posts: 321
Registered: ‎08-28-2009

Re: Tips for tweaking debug logging?

[ Edited ]

bjulin

 

to your specific questions

1) Is there any way to allocate larger (potentially very large) logging buffers on the controller?  Would that involve a reboot?

 

no, there is not. You should setup offboard syslog anyways, because even if the problem is internal to the control path, you may get what you need in the lead up to the problem. Further, the absence of logs during the problem is also another datapoint that can be used. Having said that, if you are concerned about the control paths ability to send out UDP packets, likely there are worse things going on (high dp cpu util, dp bwm issues, out of buffers etc.) and syslog from applications is unlikely to show much up.

 

 

2) Could someone explain how to use "logging level X ap-debug" since we cannot seem to find documentation for the required "name" parameter (online help says "Debug String Range 1-63")?

 

the value that needs to be input is the AP name, example

logging level debugging ap-debug  "AP name with spaces needs quotes"

logging level debugging ap-debug   LobbyAP123

 

regards

-jeff

Super Contributor I
Posts: 274
Registered: ‎04-04-2014

Re: Tips for tweaking debug logging?

 

Thanks jgoff.  We do indeed have offboard syslog also set up.

 

As far as the AP names, what I'm after is whether there is a pattern-matching thing going on with the name parameter,

or can it take a list, or anything other than just single names, and if so, if the syntax is documented?  Since I'm trying to reduce,

rather than increase, debug messages, I'd need to apply this to all APs.

 

 

 

Moderator
Posts: 321
Registered: ‎08-28-2009

Re: Tips for tweaking debug logging?

[ Edited ]

 

It's not a regexp and it doesn't take a list. The only option is to enter it multiple times. However be careful with that command without any filters at the end, by default it makes a huge amount of debug due to it picking up L2 bcast frames of all sorts and frequent logging of AM stats.

 

 

Jul 21 12:42:20 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 22 22
Jul 21 12:42:20 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Parsed ENET Frame: src=00:0d:ed:34:42:98 Dst=01:00:0c:cc:cc:cc
Jul 21 12:42:20 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: eif_add_mac: process_eif_packet MAC = 00:0d:ed:34:42:98, IP = 0.0.0.0
Jul 21 12:44:36 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 806 806
Jul 21 12:44:36 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Parsed ARP Frame: src=00:0c:c3:e7:8e:48 Dst=00:00:00:00:00:00
Jul 21 12:44:36 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: eif_add_mac: process_eif_packet MAC = 00:0c:c3:e7:8e:48, IP = 192.168.1.254
Jul 21 12:44:42 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: wifi0: MPPS 782 CPPS 23 PKTS 194706 BYTES 33503105 INTR 132159
Jul 21 12:44:42 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: wifi1: MPPS 0 CPPS 0 PKTS 0 BYTES 0 INTR 0
Jul 21 12:44:43 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 806 806
Jul 21 12:44:43 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Parsed ARP Frame: src=00:0b:86:62:0b:60 Dst=00:00:00:00:00:00
Jul 21 12:44:43 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: eif_add_mac: process_eif_packet MAC = 00:0b:86:62:0b:60, IP = 192.168.1.156
Jul 21 12:44:44 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 806 806
Jul 21 12:44:44 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Parsed ARP Frame: src=00:0b:86:62:0b:60 Dst=00:00:00:00:00:00
Jul 21 12:44:44 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: eif_add_mac: process_eif_packet MAC = 00:0b:86:62:0b:60, IP = 192.168.1.156
Jul 21 12:44:45 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 806 806

 

When using 'ap-debug' , if the issue is related to a specific module (say configuration) you can start with the process options for the debug and see if that can avoid capturing all this noise from sapd/AM.

 

example:  logging level debugging ap-debug the-ap process cfgm

 

I use cfgm as an example here, as it's probably the most common debug flag that would be used with ap-debug in terms of trying to find out a problem (dirty flag, reg domain, config seeming not to be pushed).

 

regards

-jeff

 

Search Airheads
Showing results for 
Search instead for 
Did you mean: