Wireless Access

last person joined: 6 hours ago 

Access network design for branch, remote, outdoor and campus locations with Aruba access points, and mobility controllers.
Expand all | Collapse all

Changing Controller-IP in AOS 8 with Mobility Conductor (formerly Mobility Master)

This thread has been viewed 92 times
  • 1.  Changing Controller-IP in AOS 8 with Mobility Conductor (formerly Mobility Master)

    Posted Apr 12, 2021 02:57 PM
      |   view attached

    Things to Consider and Take Care Of Before Executing the Change

    • Before doing this, make sure you have backups of your controller configs just in case.
    • Verify you can ssh to the controllers directly and login with admin account or RADIUS/TACACS+ if you've configured that.
    • You will need to update ClearPass or whatever your TACACS+/RADIUS server is ahead of time with the new IP address for the Controllers.
      • For ClearPass this is under Configuration->Network->Devices. Don't forget to add the new VRRP-IP address you plan to use under the Services->Clusters in step 12d below.
    • I would also recommend having console access if possible.
    • You can add both of the controllers new IPs to the Mobility Master/Conductor ahead of time. See step 1 below
    • Confirm controllers are L2 connected ahead of time by either sshing directly to the controllers or SSHing to Mobility Conductor and issuing "logon <ip address>" and then running "show lc-cluster group-membership". It should display l2-connected.
    • I would plan for this to be an outage. The APs are going to end up rebooting with pulling controllers out of the cluster, creating a new cluster, and adding them back in. Discussed in step 11 below.
    • If changing subnets, you will also need to update your ip default-gateway. This is taken care of in step 5 below.
    • Update the controller discovery mechanism used by the APs with the new IP address if you are using DNS or DHCP for discovery. If you do not have HA/VRRP configured for controller discovery configured be sure to add the IP address of all of your controllers.
    Steps for Executing the Change
    1.) In Mobility Master/Conductor add the new controller IP to the list of controllers for each, with IPSec Key. You can add all of them in advanced.
    1a.) Leave Authentication as IPsec Key, type in new IP address, type in the IPsec Key and Retype IPsec key (this is an arbitrary key, you will match it on the controllers in step 4)->submit
    2.) To begin this change, first, remove the first controller from cluster.
    2a.) Remove first controller from the cluster configuration at the Controller level of the hierarchy
    2a.i.) Click on the Controller name in the hierarchy on the left, then Configuration->Services->Clusters
    2a.ii.) Set Cluster group-membership to None
    2a.iii.) Submit, then Pending Changes, then Deploy Changes
    2b.) Then delete from the group subfolder level
    2b.i.) Click on the group folder in the hierarchy on the left, then Configuration->Services->Clusters
    2b.ii) Select the Cluster Name
    2b.iii.) In the Basic window that opens below, select the controller you just removed above and then click the trash can icon to delete it
    2b.iv.) Submit, then Pending Changes, then Deploy Changes
    3.) Second, change the Controller-IP Vlan for the first controller
    3a.) SSH to mobility master/conductor
    3b.) cd <name of the controller> ("cd ?" may help identify the name of the controller)
    3c.) "config t"
    3d.) "controller-ip vlan <new vlan id>", hit y, don't write mem yet
    4.) Third update masterip (mobility master/conductor IP address) with new source interface
    4a.)while still in the same CLI context as above, "masterip <mobility-conductor ip> ipsec <ipsec key> interface vlan <new vlan id>", type y, don't write mem yet
    5.) (Optional If you are changing IP address to a new subnet) Change Default Gateway to new subnet
    5a.) while still in the same CLI context as above, "no ip default-gateway <old gateway address>"
    5b.) "ip default-gateway <new gateway address>" don't write mem yet
    6.) (Optional) Change Hostname of controller if desired
    6a.) while in the same context as above, "hostname <new hostname>"
    7.) Verify config changes before committing them
    7a.) "show configuration pending"
    8.) "Write Mem" and "exit" out of config t (write mem will generate a reboot in several seconds to a few minutes, be patient)
    9.) Setup ping to old and new IP addresses and go to console of controller if available. Wait until controller is back up. This will take between 5-10 minutes.
            10.) Verify its using new IP Address and that the IPSec communication with the Mobility Conductor is working
            10a.) via Console, SSH, or MDconnect/logon to the new IP address of the controller, issue the following commands to verify configuration took:
            10a.i) "show switches" It may initially show UPDATE REQUIRED, if everything is working, this should eventually transition to UPDATE SUCCESSFUL after several seconds to a few minutes, if doesn't jump to 10.b.iii or after step 14
            10a.ii.) "show controller-ip" or "show run | inc controller-ip"
            10.a.iii.) "show run | inc masterip"
            10b.) on the Mobility Conductor CLI
            10b.i.) "exit" out of config t if you haven't already
            10b.ii.) "show switches"
            10b.iii.) if it's still showing the old IP and down:
            10b.iii.1.) Log into the UI, go to the Mobility Master/Conductor hierarchy, (The highest one) and click Controllers->New IP address->retype the IPsec Key and Retype IPsec key->submit
            10b.iii.2.) Wait about a minute or two and then do the "show switches" again in the CLI. It should show up with the new IP address and the Configuration State should finally transition to UPDATE SUCCESSFUL like the others
            11.) Once "show switches" on the Mobility Conductor shows the new IP and UPDATE SUCCESSFUL, you can move on to adding the controller to the Cluster. You will likely have to create a new Cluster, especially if you are using or plan to use VRRP in the clustering context for RADIUS communication with a NAC. This is because VRRP requires the same vlan and are changing vlans, so one controller will be in the new vlan and the other controller will be in the old vlan until both are moved.
            12.) Create new Cluster
            12a.) In the UI, click the group sub-folder under Managed Networks where the controller resides
            12b.) Click Services, the first page that comes up with be the Clusters Tab. Click the Plus sign in the bottom left of the Clusters box
            12c.) Give the new cluster a name, then hit the plus sign in the bottom left of the Controllers box
            12d.) Select the new IP address, select the group, add your new VRRP-IP and the new VLAN, hit ok, submit, then click Pending Changes and Deploy Changes
            13.) Add the new controller to the new Cluster Profile
            13a.) Click to the controller under the group hierarchy on the left hand side, it should take you straight to Services->Clusters->Cluster Profile
            13b.) Select the new Cluster Profile in the drop down menu
            13c.) Submit, Pending Changes, Deploy Changes
            14.) Repeat steps above for the second controller, Don't forget to move the console connection!

                 

                If you run into issues where one of the controllers gets rolled-back to original config, but Mobility Master/Conductor still has the new config, you can fix it on the controller, from CLI with the following:

                !!!!!DON'T DO ANYTHING ELSE IN DISASTER-RECOVERY MODE. THIS IS A GOOD WAY TO BREAK MOBILITY MASTER/CONDUCTOR!!!!!!

                 

                (MC2) #disaster-recovery on

                 

                *******************************

                Entering disaster recovery mode

                *******************************

                (DR-Mode) [mm] #configure t

                Enter Configuration commands, one per line. End with CNTL/Z

                 

                (DR-Mode) [mm] (config) #controller-ip vlan 2

                Error: interface VLAN 2 has static ip address configured at group level /mm

                 

                (DR-Mode) [mm] (config) #cd /mm/mynode

                (DR-Mode) [mynode] (config) #controller-ip vlan 2

                This configuration change requires a reboot. Do you want to continue[y/n]:y

                 

                Mobility Master will be rebooted on write memory.

                 

                (DR-Mode) ^[mynode] (config) #cd /mm

                (DR-Mode) ^[mm] (config) #masterip 192.168.2.5 ipsec Aruba123 interface vlan 2

                Change in the masterip configuration requires device to reload. Make sure the modified configuration ensures connectivity to the Master. Do you want to continue [y/n]: y

                (DR-Mode) ^[mm] (config) #no ip default-gateway 192.168.100.1

                (DR-Mode) ^[mm] (config) #ip default-gateway 192.168.2.1

                (DR-Mode) ^[mm] (config) #exit

                (DR-Mode) ^[mm] #write mem

                 

                Saving Configuration...

                 

                Configuration Saved.

                (DR-Mode) ^[mm] #

                 

                [22:16:20]:Starting rebootme

                [22:16:20]:Shutdown processing started

                 

                Because we rebooted, we didn't need to turn disaster-recovery off.  If we hadn't rebooted, you can turn DR off with the command "disaster-recovery off"



                ------------------------------
                Jeremy Bradrick
                ------------------------------

                Attachment(s)



              1. 2.  RE: Changing Controller-IP in AOS 8 with Mobility Conductor (formerly Mobility Master)

                Posted Apr 24, 2021 10:44 PM
                Or, just "write erase all" and give the controllers their new ip addresses via the console.  You don't have to define the ip addresses of controllers via the config on the Mobility Conductor;  When you write erase all and join them, the MM will just take what is assigned at the console.

                ------------------------------
                Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
                ------------------------------



              2. 3.  RE: Changing Controller-IP in AOS 8 with Mobility Conductor (formerly Mobility Master)

                Posted Apr 26, 2021 05:47 PM
                Colin,

                I don't believe this will work, unless Im doing something wrong. I just did a "write erase all" on one of my controllers and changed its IP address and subnet. The controller could not ping the MCdr until I added the controller and its PSK to the MCdr. Immediately it started pinging. 

                Also, since I hadn't updated the configuration in the MCdr with the new IP address of the controller, when the controller started talking to the MCdr, the MCdr rolled its configuration change back to its old IP address.

                ------------------------------
                Jeremy Bradrick
                ------------------------------



              3. 4.  RE: Changing Controller-IP in AOS 8 with Mobility Conductor (formerly Mobility Master)

                Posted Apr 26, 2021 08:02 PM
                You do have to remove the controller from the config on the MM, and then it will prompt you to write erase all,  Or, write erase all and remove the config from the MM.  Otherwise, the MM will try to reach the controller over the nonexistent ipsec connection between it and the MD.

                ------------------------------
                Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
                ------------------------------



              4. 5.  RE: Changing Controller-IP in AOS 8 with Mobility Conductor (formerly Mobility Master)

                Posted Apr 26, 2021 09:13 PM
                Since you are so specific in your initial post, let me be specific:

                If you delete a controller from the MM, it will make the MD reboot after confirming your entry and write erase all the MD.  All of the controller-specific configuration like port channels, individual interface configuration, hostname, ip address, etc will be gone.  All of the WLAN specific stuff defined at the container above will be intact.  Before deleting the controller, you can CD <controller name> on the MM and then type "show configuration committed" to see and possibly back up the controller-specific settings to a text file.

                You would see something like this:

                (ArubaMM-VA) [20:4c:03:1a:42:74] #show configuration committed 
                masterip 192.168.1.25 ipsec password1 interface vlan 1 
                controller-ip vlan 1 
                vlan 1 
                !
                interface gigabitethernet 0/0/0 
                    trusted 
                    trusted vlan 1-4094 
                !
                interface gigabitethernet 0/0/1 
                !
                interface gigabitethernet 0/0/2 
                !
                interface gigabitethernet 0/0/3 
                !
                interface port-channel 0 
                !
                interface port-channel 1 
                !
                interface port-channel 2 
                !
                interface port-channel 3 
                !
                interface port-channel 4 
                !                                                  
                interface port-channel 5 
                !
                interface port-channel 6 
                !
                interface port-channel 7 
                !
                interface vlan 1 
                    ip address 192.168.1.32 255.255.255.0 
                    ip igmp proxy gigabitethernet 0/0/0 
                !
                ip default-gateway 192.168.1.1 
                mgmt-user admin root ******************** 
                firewall 
                    cp-bandwidth-contract trusted-ucast 65535 
                    cp-bandwidth-contract trusted-mcast 3906 
                    cp-bandwidth-contract untrusted-ucast 9765 
                    cp-bandwidth-contract untrusted-mcast 3906 
                    cp-bandwidth-contract route 976 
                    cp-bandwidth-contract sessmirr 976 
                    cp-bandwidth-contract vrrp 512 
                    cp-bandwidth-contract arp-traffic 3906 
                    cp-bandwidth-contract l2-other 1953 
                !                                                  
                ip name-server 8.8.8.8 
                
                logging security process dot1x-proc level informational 
                logging security process authmgr level informational 
                logging user level debugging 
                hostname This-Controller 
                clock timezone America/Chicago 
                lc-cluster group-membership group1 
                country US ​
                After copying that to a text file, you can delete the controller from the GUI and the controller will reboot.  After it boots, you will have to run the initial script from the console and you can give it whatever ip address, subnet mask, default gateway, VLAN you want. 

                If you use the PSK with IP option in the initial script, the controller will connect to the MM even without adding an entry for that controller on the MM.  After going through the startup script and when it has contacted the MM successfully when you type "show switches debug" on the MM the MD will have a UNK (unknown) designation that would be waiting for you to add the mac address and model number to  a folder in the MM GUI (or add it on the commandline (configuration device de:ad:be:ef:ca:fe device-model A7280 /md/Controllers).

                After you add it, you can CD to the controller folder and then paste in the non-ip portion of the configuration that you backed up by using "show configuration committed"

                ------------------------------
                Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
                ------------------------------



              5. 6.  RE: Changing Controller-IP in AOS 8 with Mobility Conductor (formerly Mobility Master)

                Posted Apr 27, 2021 01:58 PM
                Ah that makes sense. I foresee a problem with the cluster configuration and VRRP however, at least if you are changing subnets. When I tried to change the IP address on one controller in the cluster-profile, through the GUI, to an IP in another subnet, it wouldn't let me because the VRRP-IPs weren't in the same subnet. I had to create a new cluster-profile and move the controllers to the new cluster-profile one at a time.

                I should update the document and post to reflect that new cluster-profile is only needed if the subnet is changing, which was the customer case for which I created this document originally.

                ------------------------------
                Jeremy Bradrick
                ------------------------------



              6. 7.  RE: Changing Controller-IP in AOS 8 with Mobility Conductor (formerly Mobility Master)

                Posted Apr 27, 2021 04:03 PM
                If you are changing subnets of all controllers in the cluster, the controllers need to ALL be removed from the cluster profile (at the MD level and the cluster level), before doing any of this, otherwise, yes there will be a conflict.v You would then delete the controller(s) from the MM and bring them up one at a time.  You would then be able to add them to the existing cluster, as long as the cluster doesn't have any controllers in it to start.

                Essentially, if you have a maintenance window, anyway, delete the controllers from the cluster, delete the controllers, (It will do a write erase all and a reboot) add them back using the initial console script.  There is definitely a benefit to being able to do this entirely over IP, but the console and write erase all minimizes alot of the headaches with trying to change controller-ips of controllers live.

                That is not to say that your document is not useful; others would be well advised to observe the gotchas you point out in your document.

                ------------------------------
                Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
                ------------------------------