I added the following configuration everything seems to be working. I can ping every where form VM in DC3, I can ping DC3 VM from other VMs.
Failover to DC3, when I shutdown a interface on Mikrotik, takes about 30 seconds. It is expected because Microtik does not support bfd for now.
Original Message:
Sent: Mar 03, 2023 07:08 AM
From: thomasbnc
Subject: Routing off of VXLAN to expternal router.
> Do I understand it correctly that to achieve this I need to add a secondary transit vlan on the VSX ISL link and SVIs in the production VRF?
Yes, you need a transit VLAN per VRF in use for L3 routing between the two VSX nodes in a clean design. However, you may also use one of the VLAN interfaces with an IP on both nodes (you typically have 1 or more vlan interfaces in the VRF). Technically this does the same job but is a bit of a misuse.So the recommendation clearly is a separate transit VLAN.
> If I use ospf it does not need to know anything about the other dc, right?
Nope, only between the two VSX node in the same cluster. The inter-DC connectivity is handled by BGP-EVPN (with ipv4 address-family activated)
> I am not sure how to use BGP for that. Just and iBGP peering in the same AS as the Underlay using the additional addresses?
A single Aruba switch/router only can have one AS number assigned to it. In your case it is 65001. This automatically results in an iBGP type peering between the two VSX node if you configure a neighborship in the VRF "prod". But there is nothing against it. So just configure a transit VLAN, activate a neighborship within the VRF on it and add it as network. If everything is "normal", there shouldn't be any route in your routing table coming from that peering. However, if you for instance shutdown OSPF on one of the VSX nodes and therefore lose the connectivity to the other VTEPs you still have "a way out" by routing the traffic to the other VSX node.
Original Message:
Sent: Mar 02, 2023 09:55 AM
From: tommyd
Subject: Routing off of VXLAN to expternal router.
> Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.
Do I understand it correctly that to achieve this I need to add a secondary transit vlan on the VSX ISL link and SVIs in the production VRF?
If I use ospf it does not need to know anything about the other dc, right?
I am not sure how to use BGP for that. Just and iBGP peering in the same AS as the Underlay using the additional addresses?
PS. The lack of proper quoting and threading in this forum is at least inconvenient.
Original Message:
Sent: Mar 01, 2023 07:30 AM
From: thomasbnc
Subject: Routing off of VXLAN to expternal router.
Okay, that's looking quite good in my eyes.
Regarding the prefix behind the Mikrotik router (10.180.10.0/24), this is present on all the 4 VSX devices either as eBGP route or as BGP-EVPN route (type 5) depending on the switch. And the default gateway towards the firewall is present as well on all 4 Aruba cores.
This leaves me with the question, what issue you still have. How do you test whether communication from a network attached to the core with hosts in net 10.180.10.0/24 works or not? Do you have a machine connected to for instance DC2-S2? Or do you use a ping (send us the exact command) from the CLI of DC2-S2? What is the source and the destination?
You also need to make sure that the other end knows about the networks behind your Aruba cores. Did you check? Are prefixes like 10.100.1.0/24, 10.200.1.0/24 and 10.40.1.0/24 present? Otherwise the forwarding towards the network works but the reverse path is unknown/wrong.
Another thing which may be connected to post 11 by vincent.giles. It's usually good practice to have an "IGP continuity" peering between the two VSX nodes per VRF to exchange routes in case BGP/EVPN peerings brake for a reason on one node. This is more important in a leave-spine topology than with your full-mesh network but still something you perhaps want to have. Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.
Regards,
Thomas
Original Message:
Sent: Mar 01, 2023 05:26 AM
From: tommyd
Subject: Routing off of VXLAN to expternal router.
I forgot to reply about iBGO peering. On all switches I have very simple config for now an each switch has something like this:
router bgp 65001
! underlay neighbors
address-family l2vpn evpn
! underlay neighbors activate and options
exit-address-family
!
vrf prod
! overlay external neighbors
address-family ipv4 unicast
! overlay external neighbors activate and options
redistribute connected
redistribute static
exit-address-family
I've attached full configs in the reply 7
Original Message:
Sent: Feb 28, 2023 10:16 AM
From: thomasbnc
Subject: Routing off of VXLAN to expternal router.
Hi
Thanks for the additional information.
Any reason why you have different route-target import/export statements in DC1 and DC2?
For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?
Could you please send us the extract of "show ip route vrf prod" of each of the core devices?
If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.
Regards,
Thomas
Original Message:
Sent: Feb 28, 2023 08:27 AM
From: tommyd
Subject: Routing off of VXLAN to expternal router.
Thank you for the reply.
I've attached output from show bgp l2vpn evpn
as files for clarity. How do I configure this iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network. Or can I use those propagated by evpn?
I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2. All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2.
For the interested I have attached also, somewhat cleaned configs.
Original Message:
Sent: Feb 28, 2023 06:49 AM
From: vincent.giles
Subject: Routing off of VXLAN to expternal router.
Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.
As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.
Hope this helps.
Original Message:
Sent: Feb 28, 2023 06:38 AM
From: vincent.giles
Subject: Routing off of VXLAN to expternal router.
Original Message:
Sent: Feb 27, 2023 05:56 AM
From: tommyd
Subject: Routing off of VXLAN to expternal router.
Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.
As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.
Hope this helps.
I am trying to configure two VSX pairs of Aruba-CX switches for two small DC-s which are supposed to work as a primary and backup site and host VMWare clusters. I am using this guide: https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/vxlan.pdf
The EVPN part works for me for VLANs defined locally on VSX pair as well as stretched across both clusters and devices connected over MLAG. I have problem with addresses/routes present only on one (primary) switch in eatchpair.
.Simplified diagram below.
When I try to connect external router in production VRF (DC3-R1) to one switch in each pair using BGP. I can't communicate with the remote router and ntworks behind it from secondary device in VXS pair. Routes for network behind DC3-R1 are present in ip routing table as well as type 5 routes in l2vpn table.
I do not have more links to DC3 (and interfaces) to connect all 4 switches.
Is there any solution for this problem simpler than running BGP in production VRF. Do I miss something important?
I tried to use vlan interfaces on both switches in a pair and bgp sessions to DC3-R1 and it seemed to be working but I'd like to avoid this because of complexity, there will be way more external routers connected.