Howto: Real-world Migration from Comware 5900 IRF Pair to ArubaOS-CX 8320 VSX Pair
a month ago - last edited 4 weeks ago
I installed an IRF pair of Comware 5900CP switches in the Sydney Solution Centre (SSC) many years ago. It was about time they were retired, and replaced with the latest and greatest ArubaOS-CX platform. The old Comware 5900CP switches were replaced with a pair of CX 8320 switches, running VSX. VSX provides multi-chassis LAG functionality, but with a dual control-plane enabling separated FW updates and live-update. Everything is dual-connected so that the loss of a switch will not cause an outage (such as when doing FW updates).
This pair of switches is the core of the SSC and CIC (Customer Innovation Centre). This environment includes 100s of devices, encompassing the whole HPE portfolio with servers, storage and networking.
Part 1 of the larger migration story is told in Howto: Real-world Migration from ProCurve 8206 to ArubaOS-Switch 5406R .
Compared with the previous migration which was a big bang approach, this was a gradual process. I had enough space in the rack to mount both new CX switches close to the existing Comware switches. They are close enough that existing cables can be reconnected to the new switches with minimal fuss.
That enabled me configure the old and new concurrently. As I remove L3 connections from the old, I can add them to the 8320 pair. Likewise with the physical connections, I can move over one pair at a time.
That also allowed me to shuffle transceivers and DACs that would not work in the 8320s.
Doing this gradual process meant it was easier to troubleshoot, and easy to step back if something unexpected happened or a test failed.
- Backup the current core config, and capture some of the key system data (see Config Analysis)
- Work through that config and identify any potentially difficult or impossible features and functions that will need to be modified or worked around
- Determine which global components need to be added first.
Determine the order of migration for remaining connections and features
- Plan the new IP addresses, remembering that you need 3 IP addresses for most subnets: primary, secondary and active gateway. I am using .21 and .22 for primary and secondary, with the active gateway usually .1
Make sure you do the appropriate change controls!
Analyse the existing switch config. Some of these Comware commands will be useful:
display current display int brief display ip int brief display lldp neighbor list display ip route display vlan display stp
There was really only one capability used by the Comware 5900 that the CX 8320 could not do - GVRP/MVRP. At the moment, this is mutually exclusive with VSX.
- The downstream switches need to have additional config to manually assign all the VLANs that need to be transported up to the 8320s.
- All VLANs that need to carried through the switch much be defined
- Almost universally, I used "vlan trunk allowed all" to reduce config on the 8320s.
mvrp global enable mvrp gvrp-compliance enable
There was also one other incidental capability that I had been using on the Comware switches - DHCP server. This is supported on CX, but the server and relay are mutually exclusive within the same VRF. Since relay is used a fair bit, I decided to move the two DHCP pools back to the 5406R.
No Longer Required
These DC features are no longer required (I think they were only for testing anyway).
traffic classifier DCBX operator or if-match acl 4000 # traffic behavior DCBX remark dot1p 3 # qos policy DCBX classifier DCBX behavior DCBX mode dcbx # acl number 4000 name DCBX rule 0 permit type 8906 ffff rule 5 permit type 8914 ffff
- Connect the mgmt ports for remote access
- Get a base config onto both switches
- Update the firmware to the latest
- Configure ISL and keepalive connections
- Connect the VSX pair to the existing Comware core (using a pair of QSFP+ DAC cables)
- Start migrating config and connections from the Comware.
Sample base config with ISL below:
! !Version ArubaOS-CX TL.10.04.0010 !export-password: default hostname 8320-upper banner motd ! ******************************************************** * * Sydney Solution Centre Network, HPE Aruba * * Unauthorised access is prohibited * ******************************************************** ! logrotate period monthly logrotate maxsize 20 user admin group administrators password ciphertext xxxxxx clock timezone australia/sydney ntp server 10.20.30.54 ntp enable ntp vrf mgmt ntp master vrf default stratum 2 ! ! ! snmp-server vrf mgmt snmp-server system-location Sydney Solution Centre snmp-server system-contact Richard Litchfield snmp-server community xxxxxx snmp-server community public snmp-server host 10.2.30.182 trap version v2c snmp-server host 10.2.30.186 trap version v2c ssh server vrf default ssh server vrf mgmt ! ! ! ! ! vlan 1 interface mgmt no shutdown ip static 10.2.8.21/24 default-gateway 10.2.8.1 nameserver 10.2.10.2 10.2.10.3 interface lag 256 no shutdown description ISL link for VSX no routing vlan trunk native 1 tag vlan trunk allowed all lacp mode active interface 1/1/32 no shutdown vrf attach keepalive description Heartbeat ip address 192.168.255.21/30 interface 1/1/52 no shutdown lag 256 interface 1/1/53 no shutdown lag 256 vsx system-mac 10:00:00:00:83:20 inter-switch-link lag 256 role primary keepalive peer 192.168.255.22 source 192.168.255.21 vsx-sync copp-policy dhcp-relay dhcp-server dns lldp snmp static-routes stp-global time vsx-global https-server vrf default https-server vrf mgmt
NetEdit version 2 is a great tool, and a substantial step up from version 1. It was used extensively during the migration process. I was using it as a CLI enhancement tool, adding new config to both the primary and secondary VSX members together.
The validation checks that the config is valid, and alerts you to problems and errors. The insights also come in handy for some activities.
This is a screenshot from NetEdit during the deployment phase of the changes to move the downstream 8212 connection over to the 8320s.
I was doing most of the config remotely. When making changes that may have unexpected or undesirable results (like losing connectivity), I would always use the checkpoint command to ensure that the previous system config would be restored automatically if something went wrong
checkpoint auto <minutes>
If you don't enter the confirm command within that timeframe, the previous config is applied
checkpoint auto confirm
This came in handy when changing the mgmt address from DHCP to static!
Stopping the keepalive IPs from appearing in routing tables
The IP addresses used for the keepalive link was showing up in the routing table - I didn't want that.
[syd-lan-5900cp]display ip routing-table 192.168.255.21 Summary Count : 5 Destination/Mask Proto Pre Cost NextHop Interface 0.0.0.0/0 Static 60 0 126.96.36.199 Vlan163 192.168.255.20/30 OSPF 150 25 10.2.10.21 Vlan10 10.2.10.22 Vlan10 10.2.40.21 Vlan40 10.2.40.22 Vlan40
This is simply fixed by creating a separate VRF for the keepalive (which is recommended as part of best practice anyway).
Create a new vrf called keepalive
Attach the keepalive VRF to the keepalive interfaces.
Change the VSX config to include the VRF for the keepalive
keepalive peer 192.168.255.21 source 192.168.255.22 vrf keepalive
After changing the keepalive to the new VRF, the route no longer appears in the routing table.
[syd-lan-5900cp]display ip routing-table 192.168.255.21 Summary Count : 1 Destination/Mask Proto Pre Cost NextHop Interface 0.0.0.0/0 Static 60 0 188.8.131.52 Vlan163
And the view from the 8320 GUI
The config files of the original 5900CP-IRF and the new 8320 configs are attached FYI
Richard Litchfield, HPE Aruba
Consulting System Engineer