Wired Intelligent Edge

I have 4x7220 controllers in L2 pair and CX switches connected over a routed network (same VRF though, doesn't go through firewalls or anything). After I break the routing towards one of the controllers IP address where the user was connected, CX switch doesn't switch over to another controller but the connection just dies.

I have basic UBT configuration with primary-controller and backup-controller configured, and with show ubt info I can see all 4 controller IPs there and bucket map spread evenly across all the cluster members.

After connection breaks:

Port Mac-Address Tunnel Status Gateway-Role Failure Reason
----------------------------------------------------------------------------------------------
1/1/24 b4:b6:86:26:47:28 ---/--- lan-role user unbootstrap has failed in controller

Show port-access client details show error "Status : Failed, Failed to setup User Based Tunnel"

Anyone seen something similar, or does anyone have failover working with dynamic segmentation and CX switches :) ? We had another case we we're trying to figure out when we ran into this new problem. Previously we moved controller's management cables for 2/4 controllers and tunnels broke with error "traffic is not being tunneled for registered client " for some clients. Some recovered for some reason but some were stuck. And the traffic isn't even being tunneled over the management interface... Authentication was OK.

CX 10.08.1030 and controllers are on 8.6.0.10

Please work with support on this. It seems the UAC Keepalive is every 60 seconds, so it may take up to 60 seconds before the failover will happen. That keepalive may be adjustable, but please work with TAC as what you see is not how it should be.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
------------------------------

Original Message:
Sent: Jan 13, 2022 12:34 PM
From: Jukka Aaltonen
Subject: Dynamic segmentation controller failover doesn't work

I have 4x7220 controllers in L2 pair and CX switches connected over a routed network (same VRF though, doesn't go through firewalls or anything). After I break the routing towards one of the controllers IP address where the user was connected, CX switch doesn't switch over to another controller but the connection just dies.

I have basic UBT configuration with primary-controller and backup-controller configured, and with show ubt info I can see all 4 controller IPs there and bucket map spread evenly across all the cluster members.

After connection breaks:

Port Mac-Address Tunnel Status Gateway-Role Failure Reason
----------------------------------------------------------------------------------------------
1/1/24 b4:b6:86:26:47:28 ---/--- lan-role user unbootstrap has failed in controller

Show port-access client details show error "Status : Failed, Failed to setup User Based Tunnel"

Anyone seen something similar, or does anyone have failover working with dynamic segmentation and CX switches :) ? We had another case we we're trying to figure out when we ran into this new problem. Previously we moved controller's management cables for 2/4 controllers and tunnels broke with error "traffic is not being tunneled for registered client " for some clients. Some recovered for some reason but some were stuck. And the traffic isn't even being tunneled over the management interface... Authentication was OK.

CX 10.08.1030 and controllers are on 8.6.0.10

Yep I have ticket opened. Only things you can configure under ubt are "sac-heartbeat-interval" which is 1-8 seconds and defaults to 1s, and uac-keepalive-interval which defaults to 60s but we tried to configure this to 2s. Still it does keep the user "connected" but no traffic is passed.
Original Message:
Sent: Jan 14, 2022 09:59 AM
From: Herman Robers
Subject: Dynamic segmentation controller failover doesn't work

Please work with support on this. It seems the UAC Keepalive is every 60 seconds, so it may take up to 60 seconds before the failover will happen. That keepalive may be adjustable, but please work with TAC as what you see is not how it should be.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Jan 13, 2022 12:34 PM
From: Jukka Aaltonen
Subject: Dynamic segmentation controller failover doesn't work

I have 4x7220 controllers in L2 pair and CX switches connected over a routed network (same VRF though, doesn't go through firewalls or anything). After I break the routing towards one of the controllers IP address where the user was connected, CX switch doesn't switch over to another controller but the connection just dies.

I have basic UBT configuration with primary-controller and backup-controller configured, and with show ubt info I can see all 4 controller IPs there and bucket map spread evenly across all the cluster members.

After connection breaks:

Port Mac-Address Tunnel Status Gateway-Role Failure Reason
----------------------------------------------------------------------------------------------
1/1/24 b4:b6:86:26:47:28 ---/--- lan-role user unbootstrap has failed in controller

Show port-access client details show error "Status : Failed, Failed to setup User Based Tunnel"

Anyone seen something similar, or does anyone have failover working with dynamic segmentation and CX switches :) ? We had another case we we're trying to figure out when we ran into this new problem. Previously we moved controller's management cables for 2/4 controllers and tunnels broke with error "traffic is not being tunneled for registered client " for some clients. Some recovered for some reason but some were stuck. And the traffic isn't even being tunneled over the management interface... Authentication was OK.

CX 10.08.1030 and controllers are on 8.6.0.10

Just to be sure, I shut down the port my test laptop was connected. Then I changed the UAC timeout to 1s, waited for a while and brought the port up. Then after confirming that the client was able to ping the gw (a router in DC) I changed the routing so that the controller where user was connected would be unavailable. It was just this, 3 controllers are still available from the switch. Then I forgot the whole thing and after 15 mins it still seems that the client laptop is not able to ping the gateway. So there has not been a failover to a working controller, UBT is still trying to send traffic to the controller that is not available.

It is really bad HA wise as the switch does not seem to do any kinds of keepalives but just keeps the client connected to the same controller.
Original Message:
Sent: Jan 14, 2022 09:59 AM
From: Herman Robers
Subject: Dynamic segmentation controller failover doesn't work

Please work with support on this. It seems the UAC Keepalive is every 60 seconds, so it may take up to 60 seconds before the failover will happen. That keepalive may be adjustable, but please work with TAC as what you see is not how it should be.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Jan 13, 2022 12:34 PM
From: Jukka Aaltonen
Subject: Dynamic segmentation controller failover doesn't work

I have 4x7220 controllers in L2 pair and CX switches connected over a routed network (same VRF though, doesn't go through firewalls or anything). After I break the routing towards one of the controllers IP address where the user was connected, CX switch doesn't switch over to another controller but the connection just dies.

I have basic UBT configuration with primary-controller and backup-controller configured, and with show ubt info I can see all 4 controller IPs there and bucket map spread evenly across all the cluster members.

After connection breaks:

Port Mac-Address Tunnel Status Gateway-Role Failure Reason
----------------------------------------------------------------------------------------------
1/1/24 b4:b6:86:26:47:28 ---/--- lan-role user unbootstrap has failed in controller

Show port-access client details show error "Status : Failed, Failed to setup User Based Tunnel"

Anyone seen something similar, or does anyone have failover working with dynamic segmentation and CX switches :) ? We had another case we we're trying to figure out when we ran into this new problem. Previously we moved controller's management cables for 2/4 controllers and tunnels broke with error "traffic is not being tunneled for registered client " for some clients. Some recovered for some reason but some were stuck. And the traffic isn't even being tunneled over the management interface... Authentication was OK.

CX 10.08.1030 and controllers are on 8.6.0.10

If we shut down the interfaces on the controller or remove VLAN between the controller (so that the controller understands it's clustering has failed) failover works.

However if we remove the VLAN or change the routing so that the first controller is unavailable to CX switches, but VLAN and L2 connectivity between controllers is OK, all the UBT clients on that controller fail and do not fail over to the working controller. We just see that "user unbootstrap has failed in controller" error on the CX switch but the switch does not create tunnels to the working controller. Switch can see the other controller just fine and show ubt state shows the switch has registered itself to that controller.

Show ubt info shows non-working controller incorrectly on the bucket map and on the node list though.

So far TAC hasn't had any ideas what might cause this.
Original Message:
Sent: Jan 14, 2022 11:36 AM
From: Jukka Aaltonen
Subject: Dynamic segmentation controller failover doesn't work

Just to be sure, I shut down the port my test laptop was connected. Then I changed the UAC timeout to 1s, waited for a while and brought the port up. Then after confirming that the client was able to ping the gw (a router in DC) I changed the routing so that the controller where user was connected would be unavailable. It was just this, 3 controllers are still available from the switch. Then I forgot the whole thing and after 15 mins it still seems that the client laptop is not able to ping the gateway. So there has not been a failover to a working controller, UBT is still trying to send traffic to the controller that is not available.

It is really bad HA wise as the switch does not seem to do any kinds of keepalives but just keeps the client connected to the same controller.
Original Message:
Sent: Jan 14, 2022 09:59 AM
From: Herman Robers
Subject: Dynamic segmentation controller failover doesn't work

Please work with support on this. It seems the UAC Keepalive is every 60 seconds, so it may take up to 60 seconds before the failover will happen. That keepalive may be adjustable, but please work with TAC as what you see is not how it should be.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Jan 13, 2022 12:34 PM
From: Jukka Aaltonen
Subject: Dynamic segmentation controller failover doesn't work

I have 4x7220 controllers in L2 pair and CX switches connected over a routed network (same VRF though, doesn't go through firewalls or anything). After I break the routing towards one of the controllers IP address where the user was connected, CX switch doesn't switch over to another controller but the connection just dies.

I have basic UBT configuration with primary-controller and backup-controller configured, and with show ubt info I can see all 4 controller IPs there and bucket map spread evenly across all the cluster members.

After connection breaks:

Port Mac-Address Tunnel Status Gateway-Role Failure Reason
----------------------------------------------------------------------------------------------
1/1/24 b4:b6:86:26:47:28 ---/--- lan-role user unbootstrap has failed in controller

Show port-access client details show error "Status : Failed, Failed to setup User Based Tunnel"

Anyone seen something similar, or does anyone have failover working with dynamic segmentation and CX switches :) ? We had another case we we're trying to figure out when we ran into this new problem. Previously we moved controller's management cables for 2/4 controllers and tunnels broke with error "traffic is not being tunneled for registered client " for some clients. Some recovered for some reason but some were stuck. And the traffic isn't even being tunneled over the management interface... Authentication was OK.

CX 10.08.1030 and controllers are on 8.6.0.10