Wired Intelligent Edge

I am trying to configure  two VSX pairs of Aruba-CX switches for two small DC-s which are supposed to work as a primary and backup site and host VMWare clusters. I am using this guide: https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/vxlan.pdf

The EVPN part works for me for VLANs defined locally on VSX pair as well as stretched across both clusters and devices connected over MLAG. I have problem with addresses/routes present only on one (primary) switch in eatchpair. 

.Simplified diagram below.

When I try to connect external router in production VRF (DC3-R1) to one switch in each pair using BGP. I can't communicate with the remote router and ntworks behind it from secondary device in VXS pair. Routes for network behind DC3-R1 are present in ip routing table as well as type 5 routes in l2vpn table.

I do not have more links to DC3 (and interfaces) to connect all 4 switches.

Is there any solution for this problem simpler than running BGP in production VRF. Do I miss something important?

I tried to use vlan interfaces on both switches in a pair and bgp sessions to DC3-R1 and it seemed to be working but I'd like to avoid this because of complexity, there will be way more external routers connected.

Are you using the same AS number at all the devices? If yes, then you need to make a full mesh network or use route reflectors. This because of the iBGP rules.

I am trying to configure  two VSX pairs of Aruba-CX switches for two small DC-s which are supposed to work as a primary and backup site and host VMWare clusters. I am using this guide: https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/vxlan.pdf

The EVPN part works for me for VLANs defined locally on VSX pair as well as stretched across both clusters and devices connected over MLAG. I have problem with addresses/routes present only on one (primary) switch in eatchpair. 

.Simplified diagram below.

When I try to connect external router in production VRF (DC3-R1) to one switch in each pair using BGP. I can't communicate with the remote router and ntworks behind it from secondary device in VXS pair. Routes for network behind DC3-R1 are present in ip routing table as well as type 5 routes in l2vpn table.

I do not have more links to DC3 (and interfaces) to connect all 4 switches.

Is there any solution for this problem simpler than running BGP in production VRF. Do I miss something important?

I tried to use vlan interfaces on both switches in a pair and bgp sessions to DC3-R1 and it seemed to be working but I'd like to avoid this because of complexity, there will be way more external routers connected.

I am trying to configure  two VSX pairs of Aruba-CX switches for two small DC-s which are supposed to work as a primary and backup site and host VMWare clusters. I am using this guide: https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/vxlan.pdf

The EVPN part works for me for VLANs defined locally on VSX pair as well as stretched across both clusters and devices connected over MLAG. I have problem with addresses/routes present only on one (primary) switch in eatchpair. 

.Simplified diagram below.

When I try to connect external router in production VRF (DC3-R1) to one switch in each pair using BGP. I can't communicate with the remote router and ntworks behind it from secondary device in VXS pair. Routes for network behind DC3-R1 are present in ip routing table as well as type 5 routes in l2vpn table.

I do not have more links to DC3 (and interfaces) to connect all 4 switches.

Is there any solution for this problem simpler than running BGP in production VRF. Do I miss something important?

I tried to use vlan interfaces on both switches in a pair and bgp sessions to DC3-R1 and it seemed to be working but I'd like to avoid this because of complexity, there will be way more external routers connected.

I am trying to configure  two VSX pairs of Aruba-CX switches for two small DC-s which are supposed to work as a primary and backup site and host VMWare clusters. I am using this guide: https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/vxlan.pdf

The EVPN part works for me for VLANs defined locally on VSX pair as well as stretched across both clusters and devices connected over MLAG. I have problem with addresses/routes present only on one (primary) switch in eatchpair. 

.Simplified diagram below.

When I try to connect external router in production VRF (DC3-R1) to one switch in each pair using BGP. I can't communicate with the remote router and ntworks behind it from secondary device in VXS pair. Routes for network behind DC3-R1 are present in ip routing table as well as type 5 routes in l2vpn table.

I do not have more links to DC3 (and interfaces) to connect all 4 switches.

Is there any solution for this problem simpler than running BGP in production VRF. Do I miss something important?

I tried to use vlan interfaces on both switches in a pair and bgp sessions to DC3-R1 and it seemed to be working but I'd like to avoid this because of complexity, there will be way more external routers connected.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

I am trying to configure  two VSX pairs of Aruba-CX switches for two small DC-s which are supposed to work as a primary and backup site and host VMWare clusters. I am using this guide: https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/vxlan.pdf

The EVPN part works for me for VLANs defined locally on VSX pair as well as stretched across both clusters and devices connected over MLAG. I have problem with addresses/routes present only on one (primary) switch in eatchpair. 

.Simplified diagram below.

When I try to connect external router in production VRF (DC3-R1) to one switch in each pair using BGP. I can't communicate with the remote router and ntworks behind it from secondary device in VXS pair. Routes for network behind DC3-R1 are present in ip routing table as well as type 5 routes in l2vpn table.

I do not have more links to DC3 (and interfaces) to connect all 4 switches.

Is there any solution for this problem simpler than running BGP in production VRF. Do I miss something important?

I tried to use vlan interfaces on both switches in a pair and bgp sessions to DC3-R1 and it seemed to be working but I'd like to avoid this because of complexity, there will be way more external routers connected.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

I am trying to configure  two VSX pairs of Aruba-CX switches for two small DC-s which are supposed to work as a primary and backup site and host VMWare clusters. I am using this guide: https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/vxlan.pdf

The EVPN part works for me for VLANs defined locally on VSX pair as well as stretched across both clusters and devices connected over MLAG. I have problem with addresses/routes present only on one (primary) switch in eatchpair. 

.Simplified diagram below.

When I try to connect external router in production VRF (DC3-R1) to one switch in each pair using BGP. I can't communicate with the remote router and ntworks behind it from secondary device in VXS pair. Routes for network behind DC3-R1 are present in ip routing table as well as type 5 routes in l2vpn table.

I do not have more links to DC3 (and interfaces) to connect all 4 switches.

Is there any solution for this problem simpler than running BGP in production VRF. Do I miss something important?

I tried to use vlan interfaces on both switches in a pair and bgp sessions to DC3-R1 and it seemed to be working but I'd like to avoid this because of complexity, there will be way more external routers connected.

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

I am trying to configure  two VSX pairs of Aruba-CX switches for two small DC-s which are supposed to work as a primary and backup site and host VMWare clusters. I am using this guide: https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/vxlan.pdf

The EVPN part works for me for VLANs defined locally on VSX pair as well as stretched across both clusters and devices connected over MLAG. I have problem with addresses/routes present only on one (primary) switch in eatchpair. 

.Simplified diagram below.

When I try to connect external router in production VRF (DC3-R1) to one switch in each pair using BGP. I can't communicate with the remote router and ntworks behind it from secondary device in VXS pair. Routes for network behind DC3-R1 are present in ip routing table as well as type 5 routes in l2vpn table.

I do not have more links to DC3 (and interfaces) to connect all 4 switches.

Is there any solution for this problem simpler than running BGP in production VRF. Do I miss something important?

I tried to use vlan interfaces on both switches in a pair and bgp sessions to DC3-R1 and it seemed to be working but I'd like to avoid this because of complexity, there will be way more external routers connected.

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

I am trying to configure  two VSX pairs of Aruba-CX switches for two small DC-s which are supposed to work as a primary and backup site and host VMWare clusters. I am using this guide: https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/vxlan.pdf

The EVPN part works for me for VLANs defined locally on VSX pair as well as stretched across both clusters and devices connected over MLAG. I have problem with addresses/routes present only on one (primary) switch in eatchpair. 

.Simplified diagram below.

When I try to connect external router in production VRF (DC3-R1) to one switch in each pair using BGP. I can't communicate with the remote router and ntworks behind it from secondary device in VXS pair. Routes for network behind DC3-R1 are present in ip routing table as well as type 5 routes in l2vpn table.

I do not have more links to DC3 (and interfaces) to connect all 4 switches.

Is there any solution for this problem simpler than running BGP in production VRF. Do I miss something important?

I tried to use vlan interfaces on both switches in a pair and bgp sessions to DC3-R1 and it seemed to be working but I'd like to avoid this because of complexity, there will be way more external routers connected.