Wireless Access

 View Only
last person joined: yesterday 

Access network design for branch, remote, outdoor, and campus locations with HPE Aruba Networking access points and mobility controllers.
Back to discussions
Expand all | Collapse all

APs are going up and down repeadetly

This thread has been viewed 75 times

  • 1.  APs are going up and down repeadetly

    Posted Mar 18, 2021 03:34 PM
    Hi community,

    My customer has 2 different MMs and there are two different cluster 7210 controllers under the MMs. All APs (count 525) were terminated on the first MM which is 8.6.0.6. And then i moved all APs to the new deployed MM which is 8.7.1.1 ( at same DC ). Some APs could not registered to new cluster. When i look for the logs i saw this message;

    -<399803> <3680> <ERRS> |stm| An internal system error has occurred at file sapm_ap_mgmt.c function sapm_proc_hello_req line 19376 error AP: ap-01-01 image version mismatch. bootfile is ipq806x.ari, build string is ArubaOS version 6.4.4.16 for 32x

    And then i trace the AP boot process via console, AP is trying to come up with 6.4.4.16 version and controller refused as expected. 

    These 32x APs at the remote locations and there is MPLS line between DC and locations.

    Any suggestion for this situation ?

    ------------------------------
    Tuna AKYOL #ACCA #ACMP
    ------------------------------


  • 2.  RE: APs are going up and down repeadetly

    EMPLOYEE
    Posted Mar 18, 2021 08:54 PM
    Remember that the APs need to terminate and communicate with the Mobility Controller, NOT the MM.

    ------------------------------
    Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
    ------------------------------

    Original Message


  • 3.  RE: APs are going up and down repeadetly

    Posted Mar 19, 2021 01:27 AM
    I did not say that the APs trying to communicate with MM, they already resolved controller vrrp ip address and trying to communicate with cluster.

    Thanks

    ------------------------------
    Tuna AKYOL
    ------------------------------

    Original Message


  • 4.  RE: APs are going up and down repeadetly

    EMPLOYEE
    Posted Mar 19, 2021 04:48 AM
    I would recommend reaching out to TAC Support, as NesaM sees the same. What is strange to me is that the AP is coming up with 6.4 firmware, but that would indicate that there is an issue with the 8.7.1.1 firmware after being pushed, AP will use backup firmware (6.4), connect to the controller, gets upgraded again, then fails again. What I see in other case notes is that the firmware should be 6.5 or up to allow the upgrade to 8.7, and if your backup image is 6.4 and in addition, there is a problem pushing the 8.7.1.1 version, that may result in what you see. I don't know if you can easily upgrade the backup firmware.

    TAC will be able to do a better analysis and find a solution.

    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
    ------------------------------

    Original Message


  • 5.  RE: APs are going up and down repeadetly

    Posted Mar 19, 2021 05:03 AM
    Thank you Herman, much appreciated. It would make sense that reason for it is that 6.4.4.0 (as our APs are reporting back) cannot go to AOS 8.7 directly. Do you know if some of the AP-325s come out of the box with 6.4, and some with 6.5? That would explain what we saw: out of 8 APs we tried to provision 4 did it without any issues (with S/N being almost identical (same batch?)). Anyway, clearing os didn't help, so we will have to devise some other way of doing it.


    ------------------------------
    [NesaM]
    ------------------------------

    Original Message


  • 6.  RE: APs are going up and down repeadetly

    EMPLOYEE
    Posted Mar 19, 2021 06:43 AM
    It could well be that the backup partition is different, based on when the AP was manufactured. Found that if you have the AP connected to a controller (possibly your old controller if that is still available) you can use the 'show ap image version ap-name <AP-name-here>' to check the version of the backup partition. Maybe this will help you to find out if indeed the issue is related to APs that have a 6.4 backup image only.

    In the past, there was an issue with RAP5 APs that required an upgrade of the backup partition to survive an upgrade I believe 5.x to 6.x or so. At that time, there was a command in specific controllers to upgrade the backup firmware, but that command does not work on my 8.x controllers and is mentioned in the same post not to work on 6.x either. Hopefully, TAC can provide a solution if the partition version is indeed the issue.

    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
    ------------------------------

    Original Message


  • 7.  RE: APs are going up and down repeadetly

    Posted Mar 19, 2021 04:10 AM
    Hi,

    We are having exactly the same problem (MC-7220, AP-325, AOS 8.7.1.0). TAC case opened few days ago and we are capturing output of #clear os 0 (recommended first step) this morning.
    ​

    ------------------------------
    [NesaM]
    ------------------------------

    Original Message


  • 8.  RE: APs are going up and down repeadetly

    Posted Mar 19, 2021 01:44 PM
    Latest update. Similar issues reported to Aruba Engineering (AOS-215652): AP-325 with base OS of 6.4 cannot get upgraded to AOS 8.4 and higher directly.

    The way around is to upgrade them to 8.3 (or lower) before moving them to the controller running 8.4 (or higher). This can be achieved by either pointing APs to the 8.3 controller initially (DHCP option 43), and after they upgrade change 'option 43' to 8.4+ controller. Second option is to manually upgrade AP's image via AP's console port.

    ------------------------------
    [NesaM]
    ------------------------------

    Original Message


  • 9.  RE: APs are going up and down repeadetly

    Posted Mar 19, 2021 02:14 PM
    Hi, thanks Herman for your advice and thanks NesaM.

    I tried to manually upgrade from 6.4.4.16 to 6.5.x version. But did not work anyway. So i reached out to TAC and i am waiting to escalete to other TAC. 

    I downgraded controller to 8.7.0.0 version and some of the APs were back. Maybe you would try this workaround NesaM.

    By the way there is a bug on 8.7.1.1 that isakmpd process crashes when you implement RAP with PSK. We hit that bug :) and downgraded to 8.7.0.0 and it solves our other problem(AP version mismatch).


    ------------------------------
    Tuna AKYOL
    ------------------------------

    Original Message


  • 10.  RE: APs are going up and down repeadetly

    Posted Mar 22, 2021 11:27 AM
    Hi all,

    We now have received official instructions on how to deal with this situation.

    In essence, AP-325 can come with either base OS being 6.4 or 6.5. If 6.4 it cannot be upgraded to AOS higher than 8.3, while 8.5 allows direct upgrade to (at least) 8.7.1.0.  Manual upgrade, as TA75 discovered, is not working so the only way of dealing with this is (if your WLAN is already on AOs 8.4, or higher) by adding another Mobility Controller (AOS 6.5 - 8.3) into the network and pointing APs to that one first (before repointing them to Production MDs). Hope it helps.

    ------------------------------
    [NesaM]
    ------------------------------

    Original Message


  • 11.  RE: APs are going up and down repeadetly

    Posted Mar 25, 2021 06:03 AM
    Hi everybody,

    I faced the same issue with another customer. They have 6.5 standalone controller and MM-MC 8.7.1.2 at the same time. When i tried to move AP from 6x to 8.7.1.2, it said ( on controller logs ) version mismatch. I decided to create a new VLAN and new IP block for APs. And when i moved to AP to new VLAN with new IP it successfully come up with new controller and successfully upgraded itself. 

    In my opinion, APs are holding tunnels with old controller when you tried to move AP with same IP. Maybe you can try to move APs new VLAN and try it that way.
    Hope it helps to you too :)

    ------------------------------
    Tuna AKYOL
    ------------------------------

    Original Message