Network Management

last person joined: 5 hours ago 

Keep an informative eye on your network with IMC and AirWave network management solutions.
Expand all | Collapse all

Airwave Restarting service Client Monitor Worker messages

  • 1.  Airwave Restarting service Client Monitor Worker messages

    Posted Jul 10, 2017 05:54 AM

    Hello

     

    Just had a minor blip with Airwave in that it stopped showing connected clients for a short period. After a minor panic and logging into each local controller I can see we had no break in service.

     

    Around the same time we got this message:

     

    Mon Jul 10 10:20:08 2017 System System Restarting service Client Monitor Worker

     

    Looking through the event log I can see that we get a lot of these messages, sometimes one after another. Any ideas why these crop up please?

     

    Thanks


    Alexander. 



  • 2.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Aug 30, 2017 02:07 PM

    Same here. We're getting them every 15 seconds. Screen Shot 2017-08-30 at 11.06.36 AM.png



  • 3.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Sep 21, 2017 05:37 AM

    I'm seeing these regularly, with intervals varying between a few seconds and about 15 minutes. I can't see any other events or actions that might be related to it. I've seen this both in my clean 8.2.4 installation and my previous 8.2.3. The latter has been updated several times over the years but I couldn't say which version it started on.

     

    My devices:

    - 238 switches (Cisco 3560, Aruba S1500 and Aruba 2930F)

    - 715 access points (AP-105, AP-115, AP-134, AP-135, AP-205, AP-225 and RAP3)

    - Aruba 7220 running ArubaOS 8.1.0.1.

     

    I got the Client Monitor Worker messages while I was running ArubaOS 6.x and minimal SNMP traps on the 7220 as well. As I'm unsure when the problem started, I can't say whether adding new types of switches and access points might have triggered it.

     

    As far as i know it doesn't cause any trouble, but it's filling up my event log. And I'm not too comfortable with ignoring seemingly inconsequential errors.



  • 4.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 10, 2017 09:46 PM

    I am having this problem as well and have a case open with Aruba TAC. We will be working on this more tomorrow and I will post any progress we make. Anyone else find a solution to this? I am not sure if it is related but my client counts are fluctuating every 5 minutes by 500-1000 clients and that is not normal behavior.

     

    Tue Oct 10 20:28:32 2017 System System Restarting service Client Monitor Worker
    Tue Oct 10 20:28:15 2017 System System Restarting service Client Monitor Worker
    Tue Oct 10 20:28:00 2017 System System Restarting service Client Monitor Worker
    Tue Oct 10 20:27:32 2017 System System Restarting service Client Monitor Worker
    Tue Oct 10 20:27:15 2017 System System Restarting service Client Monitor Worker
    Tue Oct 10 20:26:15 2017 System System Restarting service Client Monitor Worker
    Tue Oct 10 20:26:00 2017 System System Restarting service Client Monitor Worker
    Tue Oct 10 20:25:45 2017 System System Restarting service Client Monitor Worker
    Tue Oct 10 20:25:16 2017 System System Restarting service Client Monitor Worker 

    Capture.PNG

     

     



  • 5.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 11, 2017 04:09 AM

    I'd be really interested to hear about what you find as, as mentioned, we are getting bucket loads of these messages. 

     

    The other reason it is not great is that it means that it is obscuring the genuine messages that I'm checking the audit log for.



  • 6.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 11, 2017 01:28 PM

    The restart messages show up in the service_watcher log.  There's probably more info about the specific process if you look in the async_logger_client log (should be an option on the System -> Status page).



  • 7.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 12, 2017 12:03 PM
      |   view attached

    The async_logger_client log certainly helps in troubleshooting this. Thanks! I'm attaching a snippet of mine with MAC addresses semi-anonymized.

     

    The messages about uninitialized value in DiscoveredViaProxy.pm always come in groups of four, with two errors in line 61 followed by two errors in line 80. There seems to always be 572 lines, or 143 groups, before they stop for a few minutes. Although the recurring number of lines seemed promising I cant' make 572 or 143 fit with AP model or anything else in my system.

     

    After the first pause the <$__ANONIO__> line reference is appended, and the line number updates after each pause. When the service restarts the ANOINO reference is gone again. I haven't been able to find anything relating to this in any other logs.

    Attachment(s)

    txt
    Client Monitor Worker.txt   8 KB 1 version


  • 8.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 12, 2017 05:49 PM

    The uninit values are negligible since it's coming from device discovery.  The more concerning lines are the ones w/ 'illegal attempt to update' since the timings identical.

     

    I would probably follow a path thinking that it may be a performance issue on the system.  What's the system specs, and then number of devices monitored/managed?  Also, this is probably the right point to engage with TAC so that they can investigate further, add additional debug logging, and fine tune performance or file defect as needed.



  • 9.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 12, 2017 10:13 PM

    I have about 1750 access points and have it installed on a pretty beefy server..

     

    CPU
    Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz Hyper-Threaded 8 Cores 20480 KB cache (2600.224 MHz actual)
    Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz Hyper-Threaded 8 Cores 20480 KB cache (2600.224 MHz actual)

    Memory
    Installed Physical RAM: 62.80 GB
    Configured Swap Space: 4.00 GB



  • 10.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 16, 2017 01:04 PM

    I'm running a VM with 4 VCPUs and 24 GB RAM. AMP is using a steady 13 GB, while Kernel Cache eats up whatever's left. CPU load is pretty stable at 40% with 60% peaks about every half hour. The peaks don't coincide with Client Monitor Worker restarts.

     

    I've still got my old 8.2.3 running, with only the 240-ish switches. It still restarts CMW every few minutes a few times a day, without much detail as to why:

    Mon Oct 16 15:20:54 2017: Started (PID: 16366)
    Mon Oct 16 15:20:54 2017: Postgres PID: 16367
    Mon Oct 16 15:36:38 2017: pid 9069 exiting: reached max payload count.
    Mon Oct 16 15:36:39 2017: Started (PID: 17486)
    Mon Oct 16 15:36:39 2017: Postgres PID: 17487
    Mon Oct 16 15:39:11 2017: pid 9653 exiting: reached max payload count.
    Mon Oct 16 15:39:25 2017: Started (PID: 17706)
    Mon Oct 16 15:39:25 2017: Postgres PID: 17707
    Mon Oct 16 15:40:30 2017: pid 9656 exiting: reached max payload count.
    Mon Oct 16 15:40:39 2017: Started (PID: 17799)
    Mon Oct 16 15:40:39 2017: Postgres PID: 17806
    Mon Oct 16 15:40:59 2017: pid 9280 exiting: reached max payload count.
    Mon Oct 16 15:41:09 2017: Started (PID: 17850)
    Mon Oct 16 15:41:09 2017: Postgres PID: 17852

    It also has some illegal update attempts, albeit fewer, and seemingly not coinciding with the CMW restarts. 



  • 11.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 16, 2017 01:53 PM

    Async Logger CLient or monitoring Workers (ALC) restart after reaching certain payload or memory limit in 8.2.x is normal. Nothing to worry about. We made a change in 8.2.x for self recovering of ALC to restart if the workers reach certain threshold of payload or takes more memory to avoid memory overflow from this service.  This is hardcoded to 4G mem and 65K payload limit.

     

    So the servers are not affected by this. We will see if we can log these else where instead of event logs to avoid confuison.

     

     



  • 12.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 17, 2017 08:37 AM

    Ah, that makes sense. It's a bit worrisome that it reaches those limits every 2-10 minutes througout the day, though. Are there any known settings or functions that can cause that?

     

    It would be great if you could log it somewhere else, so it doesn't fill the event log and push out more important information.



  • 13.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 17, 2017 08:57 AM

    Agreed is this physical RAM or swap space? I've got much more than 4GB so making it configurable would be useful.

     

    Alexander

     

     



  • 14.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 17, 2017 12:20 PM

    novec - This depend son amount of AMON data that controller is sending. We need to edit in code to increase the limits but for majority of enviroments thats not needed as this entirely depends on data received per min from the network. Also depends on the server specs. Faster CPU's Faster Disk I/o etc. However, as stated not to worry about these restarts. I ahve started a thread internally to move these out of Event logs to avoid panicky assumptions.

     

    @wifialexander - this is Physical RAM. But mostly the restarts are due to size than RAM. If due to RAM, then probably the CPUs are on bit slower side. There will 2/3 number of ALC's for the numbe rof cores you have on the server. So its not one process takes up to 4G. Each ALC can take up to 4G. As before there was no need to increase this as this works for majority of envronments. If for some reason, this si needed on your server, please open a TAC case and let me know i will check what we can do on your server to make it better.



  • 15.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 18, 2017 12:03 PM

    Nice. I'm looking forward to having an event log that isn't filled up with useless information :-)



  • 16.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 12, 2017 10:09 PM
      |   view attached

    I am not sure what to make of that log at all on my system.. Attached..

    Attachment(s)



  • 17.  RE: Airwave Restarting service Client Monitor Worker messages

    Posted Oct 13, 2017 10:08 AM

    You're seeing far more issues against the association table which is a main table for data gathering.  You should definitely open a support case so that support can look into your setup further.