Wireless Access

 View Only
  • 1.  MD configuration not being inherited from MM

    Posted Feb 01, 2022 11:18 AM
    All

    Troubleshooting a problem with a Controller - all of it's config is not being inherited from the Mobility Conductor.  the vast majority of the configuraiton is, but there are a couple of items that are listed as "/local/mm/mynode" directly on the controller rather than "inherited from mm".  Very specifically, we are trying to change the VLAN that the controller is on (both the old and new VLANs are up and active on the controllers).  The only way to accomplish this is via DR mode on the controller.

    Any idea as to why this is happening nd what can be done to fix it (short of write erase all on the controller and re-pointing it at the MM)?  It is up and running now, but the client doesn't want to have to remember to go to DR mode on the controller for future changes.

    Thanks

    Jeff Johnston

    ------------------------------
    Jeffrey Johnston
    ------------------------------


  • 2.  RE: MD configuration not being inherited from MM

    Posted Feb 01, 2022 11:35 AM
      |   view attached
    Try this procedure? See below:

    • Before doing this, make sure you have backups of your controller configs just in case.
    • Verify you can ssh to the controllers directly and login with admin account or RADIUS/TACACS+ if you've configured that.
    • You will need to update ClearPass or whatever your TACACS+/RADIUS server is ahead of time with the new IP address for the Controllers.
      • For ClearPass this is under Configuration->Network->Devices. Don't forget to add the new VRRP-IP address you plan to use under the Services->Clusters in step 12d below.
    • I would also recommend having console access if possible.
    • You can add both of the controllers new IPs to the Mobility Master/Conductor ahead of time. See step 1 below
    • Confirm controllers are L2 connected ahead of time by either sshing directly to the controllers or SSHing to Mobility Conductor and issuing "logon <ip address>" and then running "show lc-cluster group-membership". It should display l2-connected.
    • I would plan for this to be an outage. The APs are going to end up rebooting with pulling controllers out of the cluster, creating a new cluster, and adding them back in. Discussed in step 11 below.
    • If changing subnets, you will also need to update your ip default-gateway. This is taken care of in step 5 below.
    • Update the controller discovery mechanism used by the APs with the new IP address if you are using DNS or DHCP for discovery. Be sure to add the IP address of both controllers.

     

     

    1. In Mobility Master/Conductor add the new controller IP to the list of controllers for each, with IPSec Key. You can add all of them in advanced.
      1. Log into the UI, go to the Mobility Master/Conductor hierarchy, (The highest one) and click Controllers->Plus sign in the bottom left->
    1. Leave Authentication as IPsec Key, type in new IP address, type in the IPsec Key and Retype IPsec key (this is an arbitrary key, you will match it on the controllers in step 4)->submit
    1. To begin this change, first, remove the first controller from cluster.
      1. Remove first controller from the cluster configuration at the Controller level of the hierarchy
    1. Click on the Controller name in the hierarchy on the left, then Configuration->Services->Clusters
    2. Set Cluster group-membership to None
    • Submit, then Pending Changes, then Deploy Changes
      1. Then delete from the group subfolder level
    1. Click on the group folder in the hierarchy on the left, then Configuration->Services->Clusters
    2. Select the Cluster Name
    • In the Basic window that opens below, select the controller you just removed above and then click the trash can icon to delete it
    1. Submit, then Pending Changes, then Deploy Changes
    1. Second, change the Controller-IP Vlan for the first controller
      1. SSH to mobility master/conductor
      2. cd <name of the controller> ("cd ?" may help identify the name of the controller)
      3. "config t"
      4. "controller-ip vlan <new vlan id>", hit y, don't write mem yet
    2. Third update masterip (mobility master/conductor IP address) with new source interface
      1. while still in the same CLI context as above, "masterip <mobility-conductor ip> ipsec <ipsec key> interface vlan <new vlan id>", type y, don't write mem yet
    3. (Optional If you are changing IP address to a new subnet) Change Default Gateway to new subnet
      1. while still in the same CLI context as above, "no ip default-gateway <old gateway address>"
      2. "ip default-gateway <new gateway address>" don't write mem yet
    4. (Optional) Change Hostname of controller if desired
      1. while in the same context as above, "hostname <new hostname>"
    5. Verify config changes before committing them
      1. "show configuration pending"
    6. "Write Mem" and "exit" out of config t (write mem will generate a reboot in several seconds to a few minutes, be patient)
    7. Setup ping to old and new IP addresses and go to console of controller if available. Wait until controller is back up. This will take between 5-10 minutes.
    8. Verify its using new IP Address and that the IPSec communication with the Mobility Conductor is working
      1. via Console, SSH, or MDconnect/logon to the new IP address of the controller, issue the following commands to verify configuration took:
    1. "show switches" It may initially show UPDATE REQUIRED, if everything is working, this should eventually transition to UPDATE SUCCESSFUL after several seconds to a few minutes, if doesn't jump to 10.b.iii or after step 14
    2. "show controller-ip" or "show run | inc controller-ip"
    • "show run | inc masterip"
      1. on the Mobility Conductor CLI
    1. "exit" out of config t if you haven't already
    2. "show switches"
    • if it's still showing the old IP and down:
      1. Log into the UI, go to the Mobility Master/Conductor hierarchy, (The highest one) and click Controllers->New IP address->retype the IPsec Key and Retype IPsec key->submit
      2. Wait about a minute or two and then do the "show switches" again in the CLI. It should show up with the new IP address and the Configuration State should finally transition to UPDATE SUCCESSFUL like the others
    1. Once "show switches" on the Mobility Conductor shows the new IP and UPDATE SUCCESSFUL, you can move on to adding the controller to the Cluster. You will likely have to create a new Cluster, especially if you are using or plan to use VRRP in the clustering context for RADIUS communication with a NAC. This is because VRRP requires the same vlan and are changing vlans, so one controller will be in the new vlan and the other controller will be in the old vlan until both are moved.
    2. Create new Cluster
      1. In the UI, click the group sub-folder under Managed Networks where the controller resides
      2. Click Services, the first page that comes up with be the Clusters Tab. Click the Plus sign in the bottom left of the Clusters box
      3. Give the new cluster a name, then hit the plus sign in the bottom left of the Controllers box
      4. Select the new IP address, select the group, add your new VRRP-IP and the new VLAN, hit ok, submit, then click Pending Changes and Deploy Changes
    3. Add the new controller to the new Cluster Profile
      1. Click to the controller under the group hierarchy on the left hand side, it should take you straight to Services->Clusters->Cluster Profile
      2. Select the new Cluster Profile in the drop down menu
      3. Submit, Pending Changes, Deploy Changes
    4. Repeat steps above for the second controller, Don't forget to move the console connection!

     

    If you run into issues where one of the controllers gets rolled-back to original config, but Mobility Master/Conductor still has the new config, you can fix it on the controller, from CLI with the following:

    !!!!!DON'T DO ANYTHING ELSE IN DISASTER-RECOVERY MODE. THIS IS A GOOD WAY TO BREAK MOBILITY MASTER/CONDUCTOR!!!!!!

     

    (MC2) #disaster-recovery on

     

    *******************************

    Entering disaster recovery mode

    *******************************

    (DR-Mode) [mm] #configure t

    Enter Configuration commands, one per line. End with CNTL/Z

     

    (DR-Mode) [mm] (config) #controller-ip vlan 2

    Error: interface VLAN 2 has static ip address configured at group level /mm

     

    (DR-Mode) [mm] (config) #cd /mm/mynode

    (DR-Mode) [mynode] (config) #controller-ip vlan 2

    This configuration change requires a reboot. Do you want to continue[y/n]:y

     

    Mobility Master will be rebooted on write memory.

     

    (DR-Mode) ^[mynode] (config) #cd /mm

    (DR-Mode) ^[mm] (config) #masterip 192.168.2.5 ipsec Aruba123 interface vlan 2

    Change in the masterip configuration requires device to reload. Make sure the modified configuration ensures connectivity to the Master. Do you want to continue [y/n]: y

    (DR-Mode) ^[mm] (config) #no ip default-gateway 192.168.100.1

    (DR-Mode) ^[mm] (config) #ip default-gateway 192.168.2.1

    (DR-Mode) ^[mm] (config) #exit

    (DR-Mode) ^[mm] #write mem

     

    Saving Configuration...

     

    Configuration Saved.

    (DR-Mode) ^[mm] #

     

    [22:16:20]:Starting rebootme

                                        (continued on next page)

    [22:16:20]:Shutdown processing started

     

    Because we rebooted, we didn't need to turn disaster-recovery off.  If we hadn't rebooted, you can turn DR off with the command "disaster-recovery off"

     



    ------------------------------
    Dustin Burns

    Lead Mobility Engineer @Worldcom Exchange, Inc.

    ACCX 1271| ACMX 509| ACSP | ACDA | MVP Guru 2022
    If my post was useful accept solution and/or give kudos
    ------------------------------

    Attachment(s)



  • 3.  RE: MD configuration not being inherited from MM

    Posted Feb 01, 2022 11:50 AM
    Thank you Dustin

    This was all done before the controllers were clustered, each was standalone at the time.  Showing the config from the controller level of the MM and everything looked good.  It's when I went directly onto the controller that you would see the issue.

    The second part (the DR part) is exactly what we had to do in order to get the controller to use the new vlan.  Everything is working fine for now.  The issue is that when I go directly to the controller via SSH (or MDC from the MM) and issue "show configuration efective detail" it shows that the config is local and not from the MM.  I'm looking for a way to push all configs from MM to MD

    ------------------------------
    Jeffrey Johnston
    ------------------------------



  • 4.  RE: MD configuration not being inherited from MM

    Posted Feb 01, 2022 12:36 PM
    Question:  Did you turn disaster recovery off?

    ------------------------------
    Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
    ------------------------------



  • 5.  RE: MD configuration not being inherited from MM

    Posted Feb 01, 2022 02:22 PM
    CJoseph - Yup - turned off DR mode.

    Found aother item - the ConductorIP parameter is also stuck on the old vlan.  Even configuring that from DR mode can't get it to change.  We're just going to write erase all to start with a fresh config and let it pull everything down from the MM.  Thanks for everyone's help

    ------------------------------
    Jeffrey Johnston
    ------------------------------



  • 6.  RE: MD configuration not being inherited from MM

    Posted Feb 01, 2022 03:48 PM
    You can try this command on the MD:

    ccm-debug full-config-sync

    ------------------------------
    Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
    ------------------------------



  • 7.  RE: MD configuration not being inherited from MM

    Posted Feb 02, 2022 06:20 AM
    CJoseph

    Unfortunately, before I saw this we went ahead and did a write erase all.  The 40 minutes it took to redo the config was easier than continued troubleshooting.

    A little more info - this client has 2 controllers and each of them were experiencing this problem to a degree.  The other controller we worked on roughly 6 weeks ago (we had to delay the second controller while the customer worked thru some issues with their FortiNac firewall) and we worked with TAC who tried that command and we ended up erasing that controller also.  This controller we had a bit more time so I was hoping to find a resolution for future use.  I was not involved in the initial config so I'm not entirely sure why this happened.

    Thanks for the additional information

    Jeff

    ------------------------------
    Jeffrey Johnston
    ------------------------------



  • 8.  RE: MD configuration not being inherited from MM

    Posted Feb 02, 2022 07:06 AM
    there are a couple of items that are listed as "/local/mm/mynode" directly on the controller rather than "inherited from mm".​

    At what level have you defined the Vlans?

    We usually define Vlan IDs & pool mapping at the cluster level and Vlan pools at a higher level.  We use pools even for single Vlans since that gives common names for multiple clusters & future flexibility if more Vlan capacity must be added.



    ------------------------------
    Bruce Osborne ACCP ACMP
    Liberty University

    The views expressed here are my personal views and not those of my employer
    ------------------------------



  • 9.  RE: MD configuration not being inherited from MM

    Posted Feb 02, 2022 09:48 AM
    The VLANs were created at the same level as the cluster is configured. But they were done before the cluster was created.

    ------------------------------
    Jeffrey Johnston
    ------------------------------