Wired Intelligent Edge

I am an HPE partner tasked with a 2 site, 4 x 8320 design/deployment for a I customer. Neither myself or HPE locally were involved in the sale so what we have ended up with isn't quite what we would have preferred. I have engaged HPE locally to answer the below question.

Short version is that we have 4 x 8320 and a customer who is expecting a lot of SFP+ ports in each site with cross-site layer 3 redundancy. There are 6 single mode pairs available for cross site connectivity.

I am already familiar with how to configure VSX in a single-site scenario using multi-chassis LAG (VSX LAG) to an edge switch, including the keepalive requirements, and I have read the 10.01 VSX guide multiple times. I have some questions around some specific scenarios we have discussed.

Scenario 1: Create 1 VSX per ste

- Is this a viable configuration (VRRP on one member in each VSX only)

- Are the links between set up as LAG or multichassis LAG? Will this need to be one LAG or two? 

Scenario 2

In this scenario I would run a single cross-site VSX pair, and treat the other switches as edge devices only. This means I could use active gateway or VRRP for layer 3 HA... 

... but  ... I now have the issue that my hosts in each site need to be dual connected for path redundancy (there is not enough fiber to cross-connect between sites), and I would be connecting to one VSX member and one edge switch with my NIC teams. The hosts do not require 802.3ad/LACP. They will be vSphere hosts with "Route Based on Originating Virtual Port" configured on each of their NIC teams.

I think this one will work, but it it's a bit "clumsy" and possibly opens itself up to human error at some later stage when I'm not around to ensure the rules are followed.

Note that all switches cascaded from either site are single path only.

Any commentary on this one would be appreciated.

Scenario 3

This one is a bit of a hybrid of the two above. I might be able to reclaim some SFP+ ports on a 5406 edge switch in one of the two sites. This would let me solve the host pathing problem at one site, while still having it at the other. Site 2 is for backup and DR, so if necessary we can consider running network teams as active/standby on this site.

Scenario 4

Failback scenario is STP + VRRP something like this. I know how to do this - but it's the last resort... and in general would be a disappointing result given the technology available.

Hi burgess,

From my first read, I would go with scenario 1. It is the cleanest one. But I'm unsure as well, on how to configure l2 redundancy between the two sites.

just to get a clear understanding, the interconnects between the two sites are l2? 

BR

Florian

Thanks Florian. Yes - intersite is Layer 2.

I would go for scenario 1 as well. I am not sure about the VRRP configuration, but purely looking at it theoretically, I would say it should work. VRRP should be able to work between two different members of different VSX clusters. VRRP should also support 1+N redundancy, so you should be able to add all 4 8320 to the VRRP instance.

I configured the interconnect between 2 VXS clusters once in an L2 scenario. I connected the two VSX clusters by using MC-LAG configuration on every cluster. 

Hi burgess,

I'm with Florian on this, the scenario 1 seems a lot cleaner.

You can create a single VSX LAG between sites with many links. If the bandwidth between sites is not that high, just 2 or 3 links would be more than enough. Note that all traffic that needs to be routed would go to the site where the VRRP is active.

This is my recommendation though:

Have you thought about having active gateway on just a pair on one site? Running VRRP between sites opens the possibility of a splitbrain. If you run AGW on a single site, you would only lose connectivity if both members of that VSX are actually shutdown.

If you need to upgrade one pair, that shouldn't be an issue at all. During the upgrade, while one member reboots the other one would give L2 access and L3 connectivity while the other one reboots.

About scenario 2, I wouldn't recommend it, how would the vSphere hosts know which link is the "active" one? Also, you are tying your hand on not having LACP on near future deployments, you would not be able to give redundancy with LACP for an "access" switch on a single site.

Regards!

Aarón

Please use scenario 1 which is the best from a topology viewpoint.

the 4 interconnect-links are part of a single VSX LAG which simplifies a lot routing. Use the same active-gateway vIP/vMAC on the 4x 8320:

the benefit is that in case of ESXi maintenance, you can move VMs without loosing MAC of default-gateway. You may see duplicates for ARP responses (which is not a big deal).

to have the gateway always on the local VSX, would it make sense to play with ACL's on the interconnect links and block the ARP entries for the active gateway? So Clients on each site will only get arp responses from the local VSX. 

Thanks. I should have pointed out that scenario 1 is my preferred option but I don't know if it is viable so I need to consider the alternatives. I'm looking to determine if this option will actually work and how to configure it. Specifically if VRRP can be configured on a single member and whether I need to configure the inter-link as LAG or VSX-LAG.

Note that the VSX documentation states the following:

-  active gateway and VRRP are mutually exclusive

- VRRP is limited to 2 members on CX-OS (8320)

Scenario 2 is viable only because active gateway would not be used.

Also Aaron - cross site layer 3 redundancy is the core requirement here. Management of split brain is not a concern (the design already includes this; I just didn't show it here).

Then you should follow Vincents solution. Two VSX pairs interconnected with a single VSX LAG and active gateway configured on all 8320s sharing virtual mac and virtual IPs.

---

@fefa2k wrote:

Then you should follow Vincents solution. Two VSX pairs interconnected with a single VSX LAG and active gateway configured on all 8320s sharing virtual mac and virtual IPs.

 

 

---

How would two VSX pairs share an active gateway configuration? 

Reminder: active-gateway is "just" a way to acheive an ARP response for downstream clients. There is no protocol. So having 2 or 3 or 4 devices

sharing the same VIP/VMAC does not have any downside beside multiple ARP responses. You may think or not filtering these ARP between VSX pairs. I'm not sure it is worth the effort considering the very little amount of traffic for ARP responses.

real IP of the same SVI extended between DCs must be of course different.

Say VSX pair 1 with CX1 & CX2. VSX pair 2 with CX3 and CX4.

Consider interface vlan 10

same active-gateway configuration on CX1, CX2, CX3, CX4.

Real IP on CX1: a.b.c.2, on CX2: a.b.c.3, on CX3: a.b.c.4, on CX4: a.b.c.5

Note that it provides active-active DC for outbound traffic (this is fine for DR/backup).

Thanks Vincent. This is what I was looking for. The VSX guide does not read this way. I'll reply again tomorrow evening after some testing.

To summarise ...

- Interconnect between VSX clusters is VSX LAG (multi-chassis lag) on both sides

- Identical active gateway config on all four switches

- acl should not be necessary on the interconnect

---

@vincent.giles wrote:

Reminder: active-gateway is "just" a way to acheive an ARP response for downstream clients. There is no protocol. So having 2 or 3 or 4 devices

sharing the same VIP/VMAC does not have any downside beside multiple ARP responses. You may think or not filtering these ARP between VSX pairs. I'm not sure it is worth the effort considering the very little amount of traffic for ARP responses.

real IP of the same SVI extended between DCs must be of course different.

Say VSX pair 1 with CX1 & CX2. VSX pair 2 with CX3 and CX4.

Consider interface vlan 10

same active-gateway configuration on CX1, CX2, CX3, CX4.

Real IP on CX1: a.b.c.2, on CX2: a.b.c.3, on CX3: a.b.c.4, on CX4: a.b.c.5

Note that it provides active-active DC for outbound traffic (this is fine for DR/backup).

---

Just want to thank everyone for their input here.

Originally I thought I was going to be forced in to one of the workaround scnearios I dicussed here because I was told by HPE locally that I could not have the same active gateway on 2 VSX pairs. Vincent set me on the right path and then about 9 hours later I received an Email with this from HPE global.

Apparently the below will be included in the 10.02 VSX guide.

Hey burgess,

Vincent can correct me if I'm wrong but I think you can already do that on version 10.01. As it was stated before, AGW has no protocol, as soon as a packet arrives to an AGW of any switch it will be forwarded so you don't need to wait for 10.02 release.

Cheers,

Aarón

---

@fefa2k wrote:

Hey burgess,

 

Vincent can correct me if I'm wrong but I think you can already do that on version 10.01. As it was stated before, AGW has no protocol, as soon as a packet arrives to an AGW of any switch it will be forwarded so you don't need to wait for 10.02 release.

 

Cheers,

 

Aarón

---

Yeah - I configured it today on 10.01 and it is all working well. What I mean is that HPE have told me that this diagram will be included in the 10.02 version of the doc.

Gents, awsome topic! i'm implementing option 1 at the moment.

but does someone know if there is a workaround for the "Maximum 16 VMACs can be configured" limitation?

The 16 VMACs is really to address very niche cases like dual-homed L3 subnet for servers for instance (having one default-route on one NIC and a specific set of route on the second NIC).

You recommend to use a unique VMAC value that is very intuitive for the network support team to identify: something like 00:00:00:00:01:01 for instance. Associated IPV4 VMAC and IPV6 VMAC must be different (in case you use v6 as well).

Remember this VMAC is always local to the SVI and no traffic is sourced from that VMAC besides ARP response and periodic hello broadcast to refresh MAC table of peer switches.

I think what Vincent is saying is that one VMAC for multiple active-gateway interfaces is fine ... other than in the dual homed situation he mentioned.

For my implementation I used the same VMAC for 40+ active-gateway interfaces, but used a different VMAC for anything on the same L2 interface as a routed firewall interface. We have some systems which need to be dual homed into the "LAN" and also with static routes to the DMZ. 

I would also go with option 1 with one caviat. VSX failover is close to hitless (<0.5 seconds - https://community.arubanetworks.com/t5/Wired-Intelligent-Edge-Campus/8320-VSF/m-p/441062/highlight/true#M3552), but does pause all flows momentarily. Depending on what is connecting to it this may or may not be an issue.

I too am about to set something up similar to this. I was thinking about using OSPF/ECMP with strict routed ports between the switches. I was going to assign those ports a static IP, add it into OSPF routing, and rely on ECMP to handle the balancing. Which in turn should remove the need for LAGs.. Thoughts?

If you need same subnet access among the 4 switches you need to carry the associated VLANs between the 2 DCs through the LAG to extend these VLANs.

And should you include or exclude these data vlans from the ISL between the VSX pairs?

They have to be inside ISL.

Thank Vincent, 

We are currently in a support case, where there is some confusion, so to be 100% clear:

1. You need to configure 1 MC-LAG on each VSX?

2. Tag all the data vlan's that cross the DC on both MC-LAG, and on both ISL's?

I'm reaching out the support to discuss this further with them.

Let me follow-up off0line. The topology is valid but the previous one you shared in your email with support is not, so better through email with all the contributors.

Let me follow-up off-line. The topology is valid but the previous one you shared in your email with support is not, so better through email with all the contributors.

Could your team update this thread with methodology of how to setup ospf, backup routes, etc... while implementing the DCI - VSX-LAG solution.   This thread made it very clear how to setup Active Gateway but seems to have gone offline when discussing routing.  Thanks

---

@nvadekar wrote: Could your team update this thread with methodology of how to setup ospf, backup routes, etc... while implementing the DCI - VSX-LAG solution.

Hi, the original purpose of this thread was not to discuss specifically about OSPF or DCI approaches so I strongly suggest you to open a whole new thread to explain what is your scenario and to describe what are your issues.

---

@nvadekar wrote: This thread made it very clear how to setup Active Gateway but seems to have gone offline when discussing routing.

---

See above.

ok, understood.  to start, can you let me know where there is further information about DCI VSX-LAG documentation, earlier in this thread there was a PPTX shared that it was suggested would be included in 10.2 docs, i checked the 10.4 docs and did not see it.   I have not been able to find any vrd or arubapedia documentation that explains best practice or reference material about this type of solution.  I would like to explore reference models, best practices about routing in this type of DCI.

thank you kindly

I thought I would mention that this threads implementation is considered a DCI solution by the DCI courseware.

Hi, 

VSX best practice (read appendix F) is available on asp and hpe:

https://support.hpe.com/hpsc/doc/public/display?docId=a00094242en_us

It is also on AFP if you're a partner.

Regarding routing, the topic could be included, but would require to cover multiple aspects leading to a long paper:

- routing protocols (ospf, bgp)

- active/passive DC pair or active/active DC pair for egress traffic: easy part

- active/passive or active/active for return traffic (ingress traffic): the complex part due to traffic hair-pinning between DC. How to manage this would be obviously outside of the scope of VSX best practice.

Nevertheless here are some options:

- use global traffic load-balancer

- /32 advertizement based on VM location (DC1 or DC2) : we have an Aruba solution for this (NAE).

I suggest to contact your local Aruba contqct, as this topic requires more than few messages in a forum thread, to analyze your requirements.

Regards,