Wired Intelligent Edge

We have 2x 6400 switches in VSX pair, one connected to primary uplink towards DC and second one to secondary. If I understood correctly using active-gateways would result traffic coming from access switches to load balance to all the uplinks so to both VSX. As I'd like to use primary VSX member only when it is active, should I use VRRP instead of active-gateways?

Uplink routers use BGP, so another question is how to configure routing between VSX switches so that primary is used is traffic goes to secondary from access switches. Do I just add a new VLAN + /31 on the ISL link and configure BGP peering between the VSX switches?

Yes, with active gateway both routers are active. VRRP is active/passive, you can use that if you want that.

I'm sorry, I'm not that firm in BGP, as I mainly work with OSPF.

But you have two BGP routers and both connected to both VSX members? 

I was thinking that maybe even if VRRP is active on the primary VSX switch, some clients just use secondary route as they see the same MAC address via that route too?

We have two BGP routers but they are only each connected to one VSX switch due to fiber runs.

There is a thing to consider with regard to what you wrote:

---

@pubjohndoe wrote: I was thinking that maybe even if VRRP is active on the primary VSX switch, some clients just use secondary route

---

That is the reason to prefer Active Gateway (active/active) over the VRRP (active/standby) for datapath efficiency...because with VRRP the traffic is still pushed over the ISL link, resulting in latency.

If the Backup VRRP Router receives traffic that must be routed across to the other Subnets, the VRRP Router bridges that traffic to the Primary VRRP Router (traversing the ISL, I add). The Primary VRRP Router then does the actual forwarding...instead with Active Gateway, both VSX Members are ready to forward Layer 3 traffic to the other Subnets individually: this method avoids the extra bridged hop from the Backup to the Primary VRRP Router.

Our primary BGP router is the one in our DC, so that would be our preferred path. Secondary VSX switch has a router connected to it that takes an extra hop to get to the DC (and to internet etc).

So we'd like all the traffic to go to primary VSX switch even if it would only use uplinks to that switch. As in any case we have BGP configured now between VSX switches and the default route is advertised from the primary VSX switch to the secondary.

So if we have 50:50 split then half traffic would go to secondary VSX, then to primary VSX, and then to the DC. Which is better path than to go out from secondary VSX to core and then have extra hop to the DC. And while we have more bandwidth between VSX pair than in the core it would be better in the first place to just get all the traffic to the primary VSX switch and have the secondary as "cold backup"

Also only subnets/VRFs we'd like to have is the "transit VRF" that connects switches to mobility controllers. We'd like to tunnel all the traffic with dynamic segmentation. Second VRF we have is for the management is it doesn't really matter what route it takes.

If I understand you scenario properly you have  2 options for south-north:

1) use VRRP instead of active-gateway. Then set the VRRP master for all SVIs on the VSX primary to make sure that the first L3 lookup happens on the VSX primary, resulting to all routed traffic to be attracted on the VSX primary. From here, the traffic would go to DC. It works but the downside is that you have traffic over the ISL including for inter-VLAN routing for traffic that does not go to DC.

2) keep using active-gateway, and rely on the BGP local-preference difference (primary setting higher LP than secondary for the routes received from the DC). In this model, the traffic hitting the VSX secondary, will be routed on the secondary, but to go to the DC, it will use the transit VLAN between secondary to primary instead of using your core link connected to the secondary. The benefit is that inter-vlan traffic is contained inside one of the VSX node without need to go to primary. 

Bottom-line, if you have very minimum inter-vlan routing both options are valid. If you have lot of inter-vlan routing (endpoint subnet1 to subnet2), then I would recommend option 2. In all cases, you have to set transit VLAN between VSX nodes for routing continuity (ex: reaching loopback of primary VSX when you loose upink from primary).

In both cases, for return traffic (north-south), you need to play with BGP attributes to advertise routes from VSX secondary with lower preference to DC. There are multiple ways of doing that depending on eBGP versus iBGP being used, AS prepending, community based routing...