Wired Intelligent Edge

Hi,

We have a EVPN VXLAN Setup using Spine/Leaf. Most of the Leafs until now have been using VSX. They perform really well, but for a new Rack we wanted something cheaper, and opted for the 6300 (24xSFP+). The connected servers are all vmware, and they do not use LACP LAGs. They just move over the MAC to the other NIC when the link goes down. The gateway is an active gateway.

I have opted to configure the two 6300 switches independently for the following reasons:

• We do not need multichassis lags
• Replacing a standalone switch is much easier then replacing a VSF Member (especially in Central)
• Upgrading VSF reboot's both members at the same time, and causes downtime. (I have seen some roadmap changing this in a future version, but upgrading both independently still seems easier)

The setup with two independent switches works well in normal circumstances. If I disconnect Cables from a server in 1 switch, all Mac addresses move instantly to the other switch and everything stays reachable. So far so good.

However when I reboot a switch the failover does not work. I see the MAC addresses of the server, and also the ARP entry pointing to the  second switch, but still I can't ping servers that used to reside on Switch 1. 

If I look at the bgp paths, I see that MAC and IP entries are present for both switches. So maybe the traffic is still sent to the old switch that is down. I can think of a couple of solutions but I'd like to hear some thoughts on this.

1. Make them into a VSF stack and accept the hassle that comes with that.
2. Use neighbor x.x.x.x  fall-over bfd. Without bfd it would not work because the switches have a default route in the underlay. (Maybe I could get rid of those though)

What would be the cleanest solution for this according to you guys? Or is there another approach that I could take here?

I would use neighbor a.b.c.d fall-over (without bfd option).

As mentioned in the VXLAN user-guide,

Using this command, when your 6300 reboot, the underlay uplink to spine should get down, the rebooting VTEP loopback should be withdraw once OSPF converged ((less than 1~2s max) and then the corresponding EVPN type-2 route with rebooting VTEP next-hop should disappear.

Hi,

We have a EVPN VXLAN Setup using Spine/Leaf. Most of the Leafs until now have been using VSX. They perform really well, but for a new Rack we wanted something cheaper, and opted for the 6300 (24xSFP+). The connected servers are all vmware, and they do not use LACP LAGs. They just move over the MAC to the other NIC when the link goes down. The gateway is an active gateway.

I have opted to configure the two 6300 switches independently for the following reasons:

• We do not need multichassis lags
• Replacing a standalone switch is much easier then replacing a VSF Member (especially in Central)
• Upgrading VSF reboot's both members at the same time, and causes downtime. (I have seen some roadmap changing this in a future version, but upgrading both independently still seems easier)

The setup with two independent switches works well in normal circumstances. If I disconnect Cables from a server in 1 switch, all Mac addresses move instantly to the other switch and everything stays reachable. So far so good.

However when I reboot a switch the failover does not work. I see the MAC addresses of the server, and also the ARP entry pointing to the  second switch, but still I can't ping servers that used to reside on Switch 1. 

If I look at the bgp paths, I see that MAC and IP entries are present for both switches. So maybe the traffic is still sent to the old switch that is down. I can think of a couple of solutions but I'd like to hear some thoughts on this.

1. Make them into a VSF stack and accept the hassle that comes with that.
2. Use neighbor x.x.x.x  fall-over bfd. Without bfd it would not work because the switches have a default route in the underlay. (Maybe I could get rid of those though)

What would be the cleanest solution for this according to you guys? Or is there another approach that I could take here?

I do have a default route in my underlay. Will this still work then? I figured that the default route will prevent the system from detecting the unreachability of the VTEP. Is this correct?

I would use neighbor a.b.c.d fall-over (without bfd option).

As mentioned in the VXLAN user-guide,

Using this command, when your 6300 reboot, the underlay uplink to spine should get down, the rebooting VTEP loopback should be withdraw once OSPF converged ((less than 1~2s max) and then the corresponding EVPN type-2 route with rebooting VTEP next-hop should disappear.

Hi,

We have a EVPN VXLAN Setup using Spine/Leaf. Most of the Leafs until now have been using VSX. They perform really well, but for a new Rack we wanted something cheaper, and opted for the 6300 (24xSFP+). The connected servers are all vmware, and they do not use LACP LAGs. They just move over the MAC to the other NIC when the link goes down. The gateway is an active gateway.

I have opted to configure the two 6300 switches independently for the following reasons:

• We do not need multichassis lags
• Replacing a standalone switch is much easier then replacing a VSF Member (especially in Central)
• Upgrading VSF reboot's both members at the same time, and causes downtime. (I have seen some roadmap changing this in a future version, but upgrading both independently still seems easier)

The setup with two independent switches works well in normal circumstances. If I disconnect Cables from a server in 1 switch, all Mac addresses move instantly to the other switch and everything stays reachable. So far so good.

However when I reboot a switch the failover does not work. I see the MAC addresses of the server, and also the ARP entry pointing to the  second switch, but still I can't ping servers that used to reside on Switch 1. 

If I look at the bgp paths, I see that MAC and IP entries are present for both switches. So maybe the traffic is still sent to the old switch that is down. I can think of a couple of solutions but I'd like to hear some thoughts on this.

1. Make them into a VSF stack and accept the hassle that comes with that.
2. Use neighbor x.x.x.x  fall-over bfd. Without bfd it would not work because the switches have a default route in the underlay. (Maybe I could get rid of those though)

What would be the cleanest solution for this according to you guys? Or is there another approach that I could take here?

For fall-over, recursive lookup of next-hop should not happen on "not exact prefix match" (i.e. /32 of the nexthop).

I do have a default route in my underlay. Will this still work then? I figured that the default route will prevent the system from detecting the unreachability of the VTEP. Is this correct?

I would use neighbor a.b.c.d fall-over (without bfd option).

As mentioned in the VXLAN user-guide,

Using this command, when your 6300 reboot, the underlay uplink to spine should get down, the rebooting VTEP loopback should be withdraw once OSPF converged ((less than 1~2s max) and then the corresponding EVPN type-2 route with rebooting VTEP next-hop should disappear.

Hi,

We have a EVPN VXLAN Setup using Spine/Leaf. Most of the Leafs until now have been using VSX. They perform really well, but for a new Rack we wanted something cheaper, and opted for the 6300 (24xSFP+). The connected servers are all vmware, and they do not use LACP LAGs. They just move over the MAC to the other NIC when the link goes down. The gateway is an active gateway.

I have opted to configure the two 6300 switches independently for the following reasons:

• We do not need multichassis lags
• Replacing a standalone switch is much easier then replacing a VSF Member (especially in Central)
• Upgrading VSF reboot's both members at the same time, and causes downtime. (I have seen some roadmap changing this in a future version, but upgrading both independently still seems easier)

The setup with two independent switches works well in normal circumstances. If I disconnect Cables from a server in 1 switch, all Mac addresses move instantly to the other switch and everything stays reachable. So far so good.

However when I reboot a switch the failover does not work. I see the MAC addresses of the server, and also the ARP entry pointing to the  second switch, but still I can't ping servers that used to reside on Switch 1. 

If I look at the bgp paths, I see that MAC and IP entries are present for both switches. So maybe the traffic is still sent to the old switch that is down. I can think of a couple of solutions but I'd like to hear some thoughts on this.

1. Make them into a VSF stack and accept the hassle that comes with that.
2. Use neighbor x.x.x.x  fall-over bfd. Without bfd it would not work because the switches have a default route in the underlay. (Maybe I could get rid of those though)

What would be the cleanest solution for this according to you guys? Or is there another approach that I could take here?

Ah, I didn't get that from the documentation.  But I've just tested it out and it works like a charm. Thanks Vincent!

For fall-over, recursive lookup of next-hop should not happen on "not exact prefix match" (i.e. /32 of the nexthop).

I do have a default route in my underlay. Will this still work then? I figured that the default route will prevent the system from detecting the unreachability of the VTEP. Is this correct?

I would use neighbor a.b.c.d fall-over (without bfd option).

As mentioned in the VXLAN user-guide,

Using this command, when your 6300 reboot, the underlay uplink to spine should get down, the rebooting VTEP loopback should be withdraw once OSPF converged ((less than 1~2s max) and then the corresponding EVPN type-2 route with rebooting VTEP next-hop should disappear.

Hi,

We have a EVPN VXLAN Setup using Spine/Leaf. Most of the Leafs until now have been using VSX. They perform really well, but for a new Rack we wanted something cheaper, and opted for the 6300 (24xSFP+). The connected servers are all vmware, and they do not use LACP LAGs. They just move over the MAC to the other NIC when the link goes down. The gateway is an active gateway.

I have opted to configure the two 6300 switches independently for the following reasons:

• We do not need multichassis lags
• Replacing a standalone switch is much easier then replacing a VSF Member (especially in Central)
• Upgrading VSF reboot's both members at the same time, and causes downtime. (I have seen some roadmap changing this in a future version, but upgrading both independently still seems easier)

The setup with two independent switches works well in normal circumstances. If I disconnect Cables from a server in 1 switch, all Mac addresses move instantly to the other switch and everything stays reachable. So far so good.

However when I reboot a switch the failover does not work. I see the MAC addresses of the server, and also the ARP entry pointing to the  second switch, but still I can't ping servers that used to reside on Switch 1. 

If I look at the bgp paths, I see that MAC and IP entries are present for both switches. So maybe the traffic is still sent to the old switch that is down. I can think of a couple of solutions but I'd like to hear some thoughts on this.

1. Make them into a VSF stack and accept the hassle that comes with that.
2. Use neighbor x.x.x.x  fall-over bfd. Without bfd it would not work because the switches have a default route in the underlay. (Maybe I could get rid of those though)

What would be the cleanest solution for this according to you guys? Or is there another approach that I could take here?