Controllerless Networks

last person joined: yesterday 

Instant Mode - the controllerless Wi-Fi solution that's easy to set up, is loaded with security and smarts, and won't break your budget
Back to discussions
Expand all | Collapse all

Aruba IAP Cluster reboots randomly

This thread has been viewed 6 times

  • 1.  Aruba IAP Cluster reboots randomly

    Posted Jul 09, 2019 10:55 AM

    Good morning to all of you airheads.

     

    One customer is having issues with a Cluster of Aruba IAPs 305. They reboot randomly. Here's an example of the debugging we did:

     

     

    2019-07-04 14:44:42.000 192.168.220.232
    2019 192.168.220.232 cli[3175]: <341014> <INFO> <192.168.220.232 24:F2:7F:C8:BF:78> AP rebooting System cmd at uptime 8D 0H 52M 37S: Gateway unreachable.
    2019-07-04 14:44:42.000 192.168.220.232
    2019 192.168.220.232 nanny[3099]: <303086> <ERRS> <192.168.220.232 24:F2:7F:C8:BF:78> Process Manager (nanny) shutting down - AP will reboot!
    2019-07-04 14:44:42.000 192.168.220.232
    2019 192.168.220.232 <192.168.220.232 24:F2:7F:C8:BF:78> sapd[3194]: SAPD received SIGTERM; exiting
    2019-07-04 14:44:42.000 192.168.220.232
    2019 192.168.220.232 cli[3175]: <341089> <ERRS> <192.168.220.232 24:F2:7F:C8:BF:78> cli_data_post_to_airwave: 10210: AWC login error.
    2019-07-04 14:44:42.000 192.168.220.232
    2019 192.168.220.232 cli[3175]: <341014> <INFO> <192.168.220.232 24:F2:7F:C8:BF:78> AP rebooting Gateway unreachable.
    2019-07-04 14:44:42.000 192.168.220.232
    2019 192.168.220.232 <192.168.220.232 24:F2:7F:C8:BF:78> KERNEL(AP-INJU-04@192.168.220.232): [694351.890735] NOHZ: local_softirq_pending 08
    2019-07-04 14:44:41.000 192.168.220.232
    2019 192.168.220.232 <192.168.220.232 24:F2:7F:C8:BF:78>

     

    Another example: 2019 192.168.220.232 cli[3175]: <341014> <INFO> <192.168.220.232 24:F2:7F:C8:BF:78> AP rebooting System cmd at uptime 0D 20H 26M 40S: Gateway unreachable.
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 nanny[3099]: <303086> <ERRS> <192.168.220.232 24:F2:7F:C8:BF:78> Process Manager (nanny) shutting down - AP will reboot!
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 stm[3205]: <304008> <DBUG> <192.168.220.232 24:F2:7F:C8:BF:78> |ap| DEL: bridge user doesn't exist ip:fe80::1022:812b:23e1:6b9 mac:00:00:00:00:00:00
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 stm[3205]: <304008> <DBUG> <192.168.220.232 24:F2:7F:C8:BF:78> |ap| DEL: bridge user doesn't exist ip:fe80::a650:46ff:fefb:96d6 mac:00:00:00:00:00:00
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 stm[3205]: <304008> <DBUG> <192.168.220.232 24:F2:7F:C8:BF:78> |ap| user ip:fe80::b4ce:e657:5b88:1a31 mac:00:00:00:00:00:00
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 stm[3205]: <304008> <DBUG> <192.168.220.232 24:F2:7F:C8:BF:78> |ap| user ip:fe80::960e:6bff:fe49:a4d3 mac:00:00:00:00:00:00
    2019-07-05 11:16:09.000 192.168.220.6
    2019 192.168.220.6 stm[3149]: <400166> <DBUG> <192.168.220.6 24:F2:7F:C8:BF:7C> wifi_ap_down_rap processing started for 24:f2:7f:0b:f7:d0
    2019-07-05 11:16:09.000 192.168.220.6
    2019 192.168.220.6 stm[3149]: <400110> <DBUG> <192.168.220.6 24:F2:7F:C8:BF:7C> AP AP-INJU-01: Delete AP state for bssid 24:f2:7f:0b:f7:d0; Deauth 1 Clear 1
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 <192.168.220.232 24:F2:7F:C8:BF:78> claritylive[3881]: clarity: Number of dns records:0
    2019-07-05 11:16:09.000 192.168.220.230
    2019 192.168.220.230 stm[3203]: <400166> <DBUG> <192.168.220.230 24:F2:7F:C8:BF:74> wifi_ap_down_rap processing started for 24:f2:7f:0b:f7:50
    2019-07-05 11:16:09.000 192.168.220.230
    2019 192.168.220.230 <192.168.220.230 24:F2:7F:C8:BF:74> stm[3203]: stm_send_sta_offline: Sending sta offline msg to CLI0, mac='a8:8e:24:1c:90:55'
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 <192.168.220.232 24:F2:7F:C8:BF:78> KERNEL(AP-INJU-04@192.168.220.232): [73600.329189] NOHZ: local_softirq_pending 08
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 cli[3175]: <341014> <INFO> <192.168.220.232 24:F2:7F:C8:BF:78> AP rebooting Gateway unreachable.
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 stm[3205]: <304008> <DBUG> <192.168.220.232 24:F2:7F:C8:BF:78> |ap| user ip:fe80::1022:812b:23e1:6b9 mac:00:00:00:00:00:00
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 stm[3205]: <304008> <DBUG> <192.168.220.232 24:F2:7F:C8:BF:78> |ap| user ip:fe80::a650:46ff:fefb:96d6 mac:00:00:00:00:00:00
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 stm[3205]: <304008> <DBUG> <192.168.220.232 24:F2:7F:C8:BF:78> |ap| user ip:169.254.13.42 mac:00:00:00:00:00:00
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 <192.168.220.232 24:F2:7F:C8:BF:78> stm[3205]: rap_bridge_user_handler: 14775: user entry deleted for '169.254.13.42' '50:3e:aa:77:c7:19'
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 stm[3205]: <304008> <DBUG> <192.168.220.232 24:F2:7F:C8:BF:78> |ap| DEL: bridge user doesn't exist ip:fe80::960e:6bff:fe49:a4d3 mac:00:00:00:00:00:00
    2019-07-05 11:16:09.000 192.168.220.232
    2019 192.168.220.232 stm[3205]: <304008> <DBUG> <192.168.220.232 24:F2:7F:C8:BF:78> |ap| rap_bridge_user_handler : failed
    2019-07-05 11:16:09.000 192.168.220.6
    2019 192.168.220.6 stm[3149]: <304008> <DBUG> <192.168.220.6 24:F2:7F:C8:BF:7C> |ap| wifi_vap_down_from_sapd ip=127.0.0.1
    2019-07-05 11:16:09.000 192.168.220.6
    2019 192.168.220.6 <192.168.220.6 24:F2:7F:C8:BF:7C> stm[3149]: sap_sta_mac_ppsk_timer_start: 18393: mac ppsk timer start
    2019-07-05 11:16:09.000 192.168.220.6
    2019 192.168.220.6 <192.168.220.6 24:F2:7F:C8:BF:7C> stm[3149]: stm_send_sta_offline: Sending sta offline msg to CLI0, mac='ac:2b:6e:94:69:ed'
    2019-07-05 11:16:09.000 192.168.220.230
    2019 192.168.220.230 stm[3203]: <304008> <DBUG> <192.168.220.230 24:F2:7F:C8:BF:74> |ap| wifi_vap_down_from_sapd ip=127.0.0.1
    2019-07-05 11:16:09.000 192.168.220.230
    2019 192.168.220.230 <192.168.220.230 24:F2:7F:C8:BF:74> stm[3203]: stm_send_sta_offline: Sending sta offline msg to CLI0, mac='ac:2b:6e:92:cb:65'
    2019-07-05 11:16:09.000 192.168.220.230
    2019 192.168.220.230 stm[3203]: <400110> <DBUG> <192.168.220.230 24:F2:7F:C8:BF:74> AP AP-INJU-02: Delete AP state for bssid 24:f2:7f:0b:f7:50; Deauth 1 Clear 1
    2019-07-05 11:16:09.000 192.168.220.230
    2019 192.168.220.230 <192.168.220.230 24:F2:7F:C8:BF:74> stm[3203]: stm_send_sta_offline: Sending sta offline msg to CLI0, mac='a8:8e:24:1c:90:55'
    2019-07-05 11:16:08.000 192.168.220.6
    2019 192.168.220.6 <192.168.220.6 24:F2:7F:C8:BF:7C> sapd[3138]: SAPD received SIGTERM; exiting
    2019-07-05 11:16:08.000 192.168.220.230
    2019 192.168.220.230 cli[3173]: <341014> <INFO> <192.168.220.230 24:F2:7F:C8:BF:74> AP rebooting Gateway unreachable.
    2019-07-05 11:16:08.000 192.168.220.6
    2019 192.168.220.6 stm[3149]: <304008> <DBUG> <192.168.220.6 24:F2:7F:C8:BF:7C> |ap| DEL: bridge user doesn't exist ip:fe80::7078:add4:4831:2826 mac:00:00:00:00:00:00
    2019-07-05 11:16:08.000 192.168.220.6
    2019 192.168.220.6 stm[3149]: <304008> <DBUG> <192.168.220.6 24:F2:7F:C8:BF:7C> |ap| user ip:fe80::746b:302a:58c9:da47 mac:00:00:00:00:00:00
    2019-07-05 11:16:08.000 192.168.220.6

     

     

    What we already did:

    1) Updated OS version to 8.3.0.6

    2) Selected another preferred master

    3) Checked power budget of switch (it's sufficient)

    4) Default gateway is a Cisco MPLS router.

     

     

    Thanks in advance and sorry for the long post!



  • 2.  RE: Aruba IAP Cluster reboots randomly

    EMPLOYEE
    Posted Jul 09, 2019 11:48 PM

    It looks like the instant cluster is losing connection to its default gateway which causes the Instant cluster to reboot.

    Is your network connectivity between IAP's and default gateway stable?

     

     



  • 3.  RE: Aruba IAP Cluster reboots randomly

    Posted Jul 11, 2019 09:54 AM

    Hello ariyap.

     

    Yesterday our client changed the default gateway to another device in the network. That way we will be able to disregard this as an issue.

    If the cluster doesn't reboot in the next week, I will let you and the community know.

    If they keep rebooting, could there be another issue?

    Thanks for your fast response.

    Gonzalo



  • 4.  RE: Aruba IAP Cluster reboots randomly

    MVP EXPERT
    Posted Jul 11, 2019 11:12 AM

    There could be numerous issues both hardware and software. Do you have a syslog server set up to capture any output prior to the AP rebooting?



  • 5.  RE: Aruba IAP Cluster reboots randomly

    Posted Jul 11, 2019 11:32 AM
      |   view attached

    Hi Craig.

    We have a syslog server, yes. 

    Attached you will find log info.

    Regards.

    Gonzalo.

    Attachment(s)

    txt
    05-07.txt   108 KB 1 version


  • 6.  RE: Aruba IAP Cluster reboots randomly

    MVP EXPERT
    Posted Jul 11, 2019 04:40 PM

    Seems to be the same issue as before, the IAP reboots due to gateway unreachable.

     

    2019 192.168.220.232 cli[3175]: <341014> <INFO> <192.168.220.232 24:F2:7F:C8:BF:78> AP rebooting System cmd at uptime 0D 20H 26M 40S: Gateway unreachable.
2019-07-05 11:16:09.000	192.168.220.232
2019 192.168.220.232 nanny[3099]: <303086> <ERRS> <192.168.220.232 24:F2:7F:C8:BF:78> Process Manager (nanny) shutting down - AP will reboot!


  • 7.  RE: Aruba IAP Cluster reboots randomly

    EMPLOYEE
    Posted Jul 11, 2019 06:48 PM

    This is the correct behaviour of an Instant Cluster, if the default gateway is unreachable from the configured uplink, it will try the other uplinks if configured, if not it will reboot.



  • 8.  RE: Aruba IAP Cluster reboots randomly

    Posted Jul 24, 2019 10:03 AM

    Good morning everyone. First of all thanks for your responses, I wasn't in the office and couldn't answer before.

     

    We changed default gateway from 192.168.220.1 to another L3 device (192.168.220.2) and they rebooted again. Same error "gateway unreachable".  

    We also removed device 192.168.220.1 and changed it for another router on Friday but the problem persist. The syslog level is on Debug, this means I can't get more detailed information about this reason "gateway unreachable". We are monitoring gateways (192.168.220.1 and 220.2) through other devices and they never fail to respond, so, it would seem an IAP issue but we can't confirm this yet.

    Is it possible to disable this reboot behaviour? 

     

    Thanks in advance.

    Best regards.

     

     

     



  • 9.  RE: Aruba IAP Cluster reboots randomly
    Best Answer

    Posted Sep 17, 2020 02:40 PM

    Changed APs and it worked ok so the problem was associated with these devices.

    Thanks