Wireless Access

We have a hard-to-diagnose problem we are working with the TAC.  We have to debug a problem that occurs only a couple of times a day, and which we cannot depend on external syslog to reliably deliver debug messages for, since it may involve switching drops on the control plane.  We are trying to capture at the debug level of logging during the problem on the controller, however that level of logging easily wraps the 75Kish system log buffer by the time we can get in and show the logs, much less tar them up.

We are currently trying to guess which of the most noisy debug level processes we can turn off and then individually turn debug on for as many other processes as we can, each by hand, since it does not appear one can override a "debug" with a more specific "info" or "errors" setting.  However, this will probably not be sufficient to prevent a log wrap on the services we are interested in.

So two specific questions:

1) Is there any way to allocate larger (potentially very large) logging buffers on the controller?  Would that involve a reboot?

2) Could someone explain how to use "logging level X ap-debug" since we cannot seem to find documentation for the required "name"

parameter (online help says "Debug String Range 1-63")?

Other than that, does anyone have any other tips or tricks that might be useful in this situation?

bjulin,

To get the specific debug message required, TAC must provide that information.  If it involves "drops" in the control plane, they need to replicate that in their lab, because they have specialized methods of obtaining that information that is not user-facing.

Unfortunately TAC has not been able to replicate the problem, so I am trying my best to get whatever information I can out of our production setup.  Any help would be appreciated.

bjulin,

If you feel that you are getting nowhere with TAC, you should escalate it.  You have not given us enough detail in your post to determine what is the issue and as a result, we cannot give you enough information to resolve it.  Please escalate with TAC.

The only thing I can think of based on the information you posted would be to write a script that copies the logs off to a tftp or scp server every X minutes.

cjoseph -- To be clear, I'm not asking for help with this particular problem.  I'm already "escalated" according to TAC and have also asked my SE to advise on the case.

My questions, above, are intended to both help me with this case and with future cases, and help anyone who happens across this thread.  So are there answers?

cappalli -- Thanks for the suggestion.  I've manage to tame the logs enough it just might be feasible to do so; we'll see once the issue occurs again.  Hopefully I haven't excluded essential messages.

Anyone know what setting controls 'nanny' logs that get enabled with "logging level debug system?"

bjulin

to your specific questions

1) Is there any way to allocate larger (potentially very large) logging buffers on the controller?  Would that involve a reboot?

no, there is not. You should setup offboard syslog anyways, because even if the problem is internal to the control path, you may get what you need in the lead up to the problem. Further, the absence of logs during the problem is also another datapoint that can be used. Having said that, if you are concerned about the control paths ability to send out UDP packets, likely there are worse things going on (high dp cpu util, dp bwm issues, out of buffers etc.) and syslog from applications is unlikely to show much up.

2) Could someone explain how to use "logging level X ap-debug" since we cannot seem to find documentation for the required "name" parameter (online help says "Debug String Range 1-63")?

the value that needs to be input is the AP name, example

logging level debugging ap-debug  "AP name with spaces needs quotes"

logging level debugging ap-debug   LobbyAP123

regards

-jeff

Thanks jgoff.  We do indeed have offboard syslog also set up.

As far as the AP names, what I'm after is whether there is a pattern-matching thing going on with the name parameter,

or can it take a list, or anything other than just single names, and if so, if the syntax is documented?  Since I'm trying to reduce,

rather than increase, debug messages, I'd need to apply this to all APs.

It's not a regexp and it doesn't take a list. The only option is to enter it multiple times. However be careful with that command without any filters at the end, by default it makes a huge amount of debug due to it picking up L2 bcast frames of all sorts and frequent logging of AM stats.

Jul 21 12:42:20 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 22 22
Jul 21 12:42:20 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Parsed ENET Frame: src=00:0d:ed:34:42:98 Dst=01:00:0c:cc:cc:cc
Jul 21 12:42:20 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: eif_add_mac: process_eif_packet MAC = 00:0d:ed:34:42:98, IP = 0.0.0.0
Jul 21 12:44:36 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 806 806
Jul 21 12:44:36 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Parsed ARP Frame: src=00:0c:c3:e7:8e:48 Dst=00:00:00:00:00:00
Jul 21 12:44:36 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: eif_add_mac: process_eif_packet MAC = 00:0c:c3:e7:8e:48, IP = 192.168.1.254
Jul 21 12:44:42 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: wifi0: MPPS 782 CPPS 23 PKTS 194706 BYTES 33503105 INTR 132159
Jul 21 12:44:42 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: wifi1: MPPS 0 CPPS 0 PKTS 0 BYTES 0 INTR 0
Jul 21 12:44:43 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 806 806
Jul 21 12:44:43 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Parsed ARP Frame: src=00:0b:86:62:0b:60 Dst=00:00:00:00:00:00
Jul 21 12:44:43 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: eif_add_mac: process_eif_packet MAC = 00:0b:86:62:0b:60, IP = 192.168.1.156
Jul 21 12:44:44 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 806 806
Jul 21 12:44:44 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Parsed ARP Frame: src=00:0b:86:62:0b:60 Dst=00:00:00:00:00:00
Jul 21 12:44:44 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: eif_add_mac: process_eif_packet MAC = 00:0b:86:62:0b:60, IP = 192.168.1.156
Jul 21 12:44:45 :326001:  <DBUG> |AP ap105-24:78@192.168.1.6 sapd|  AM: process_eif_packet: Untagged Frame Received 806 806

When using 'ap-debug' , if the issue is related to a specific module (say configuration) you can start with the process options for the debug and see if that can avoid capturing all this noise from sapd/AM.

example:  logging level debugging ap-debug the-ap process cfgm

I use cfgm as an example here, as it's probably the most common debug flag that would be used with ap-debug in terms of trying to find out a problem (dirty flag, reg domain, config seeming not to be pushed).

regards

-jeff