Wired Intelligent Edge

Hi,

This is my first post, so hello everyone.

We have a new 3 racks setup. The idea is to have 2xToR per rack + a 5406R as distribution switch.

My question is regarding availability. Does it make sense to invest in 2x5406R and use VSF or does one 5406R with two power supply and 2 management modules enough to provide redundancy and failover?

I guess that more is better (?) but does it really make sense for a small setup like this (that should be available al the time) 

thanks!

em

Interesting question.

IMHO setting up two Aruba 5406R zl2 in a VSF Stack should require a further step (at least involving any Server host eventually connected to the VSF) to unleash the full power of such type of deployment: that each connected host uses two links (2 ports Port Trunking using LACP) spanning to both VSF member Switches...this hosts<-->distribution level added redundancy will avoid that, if just one of the two distribution layer Switches goes down, you then will suffer the lost of all the hosts (solely) connected to it.

I know that is something very hard (and expensive) to achive with client hosts (single NIC equipped, two cables required to Rack per each Client) but make sense considering - at least - Servers, if any Server is eventually connected through the Distribution Switch(es) inside the Rack...well...that's more simple to do.

Having VSF and hosts connected to just one Switch only is like using the VSF "redundancy" feature at half of its real power (or just using it as a expensive way to expand "horizontally" your network, which could be the case anyway, depending on available budget): I mean, that's great...but...what will happen if/when one Switch of the VSF Stack goes down? say goodbye to its directly and solely connected hosts.

More or less that is the same problem one face when 2 or more Comware based Switches are depolyed in a IRF Stack: Is the IRF feature used to expand the Network horizontally only? or is it used to achieve real redundancy at hosts<-->distribution level too? or possibly to achieve both goals?

This may explain why, usually, the role of Aruba 5400R zl2 Switch Series is to act (or be placed) as Core switch (in both cases: when depoyed as single Switch and when deployed in a VSF Stack).

Said so...if your Distribution level numbers can be managed with just one Aruba 5406R zl2 (in terms of used ports involved, modules used and type of connected hosts) probably using just a single Aruba 5406R zl2 equipped with (A) redundant Power Supplies and (B) redundant Management Modules is the fastest and cheaper way to go, far less expensive than buying two identical Aruba 5406R zl2 Switches (consider that you need 10G or 40G ports for linking VSF members).

You should then be aware that if - in a near/far future - you are going to deploy VSF you will:

• (Limitation) lose the second Management Module (it must be put in Offline mode to avoid issues) on each VSF Member.
• (Requirement) required to provide/use only v3 zl2 Modules.
• (Requirement) required to made the Aruba 5400R zl2 working in v3-mode only (so loosing any v2 zl Module).
• (Requirement) required to use 10G or 40G ports for the VSF setup, possibly using (as a best practice) ports on different Modules.
• (Requirement) required to deploy a MAD Device (such as an Aruba 2920).

but, apart from these restrictions/limitations, you will gain in redundancy (protected by dual Power Supplies per unit too) and you will be able to expand your Network horizontally (growing it quite flawlessly).

Thanks for the explanation.. let me add a bit more info about my planned setup.

We will have 2 ToR on each switch.

• These ToR will be stacked; either 2xFlexFabric 5700 (with IRF) or 2xAruba3800 with stacking module.
• All servers in the racks will have 2 network cables. One cable to each ToR. We will then configure LACP on each server and Trunk/LACP on ToR.
• From each ToR we will uplink with SFP+ to our core/distribution switch(es) Aruba5406R.

Question:

•  Does this extra info help, o changes anything? :)
• If I start with a single 5406R (double power, double management) and then decide to add another 5406R, can this be done without down time (adding the second one and configuring VSF)?

Questions regarding LACP setup (can open a new Post if needed):

• Will this LACP on server side enough to expect a switch to die and continue with operation? Or do I need to also tell the ToR to LACP it's uplink?
• On the stacked ToR I configure a LACP Trunk with both ports where the 2 network cables from the server are connected. Do I also need to LACP Trunk on the 5406R (If I use two, with VSF). I believe the answer is yes, but just in case.

thanks again.

Wait just a minute and watch this presentation ([ATM16] Take a Walk On The Wired Side): there is a Tier-3 scenario considered at the end but also explains the positioning of Aruba 5400R Switch Series with respect to FlexNetwork Switches.

I think VSF, when depolyed with the Auto-Join/Plug&Play procedure, doesn't require the reboot of the first Switch but for sure requires the reboot of the second one added. On the contrary when VSF is deployed using the Manual Configuration procedure a reboot of the first Switch and then a reboot of the second one is required. Using the VSF (Loose/Strict) Provisioning procedure apparently - but I could be wrong here - requires to reboot the first Switch only...even if I'm not sure (to me this procedure will require the reboot of the second one too).

So only the Auto-Join/Plug&Play procedure will grant you no downtime IF you already have an Aruba 5400R zl2 working in v3-mode only.

If your Servers will have two aggregated links that will span to two different members of the ToR that will be OK...since the ToR's Stack (made with IRF in case of FlexNetwork Switches), from the point of view of your Servers, acts as a single virtual Switch...so LACP from Servers to the IRF will be good enough.

IMHO you will need to use redundant links (LACP) between the tiers (Distributinon and Core) in both cases:

• if you are going to use Aruba 5400R zl2 as a single or in a VSF Stack.
• if you are going to use Aruba 3800 single or (Hardware) Stacked.

I think in both cases LACP links should span - as a best practice - to both members (VSF of 5400R zl2 or Hardware Stack of 3800) but, as I understood, it will not be a particular Distributed Trunking (lacp-dt)...just a plain Trunking (lacp) between tiers...since both ends (Distribution and Core), cause the stacking, act a single virtual switches.

Probably, if I were you (you're a lucky guy with all that metal!), I would deploy two HPE FlexNetwork 5700 with IRF as ToR at Core and then I would use (Hardware) stacked Aruba 3800 Switches as distribution level..but...not definitely not Aruba 3800 as ToR at Core (using then those Aruba 3800 at Core with one or two Aruba 5400R zl2 at distribution level make no sense at all to me).

Maybe I wasn't clear about the design.

The Aruba 3800 (or FlexNetwork 5700, depending on $$) will only be for server ToR. Then they will be uplinked to 5406R via sfp+.

This uplink will end up in the 5406. 

So, at the end, the 5406 will only have the uplinks of the ToR. 

Howdy,

What I would be asking is -

             What is the underlying requirement in this "mini-datacentre" build?

A - Applications

B - Infrastructure building blocks

C - Connectivity 

D - dependencies 

How many hosts / ports / storage?

What do the traffic volumes and application spread look like today and how are they expected to ramp up?

Do you just want a Layer2 fabric with routing in the "core/aggregation" layer?

     You could just IRF all of the ToR switches together in a big ring...

Do you want a fully meshed Layer 3 routed Spine-leaf like topology?

       Very popular these days Sir. Routing right down to the edge, multiple uplinks, equal cost multi-path OSPF etc. 

Does this network need to participate with other networks either in other premises or with virtual infrastructure in a public cloud somewhere?

Will you be running Virtual switches in the servers (that we haven't yet mentioned) in the racks? 

Should your Server guy be looking at less servers but with 25Gb NICs plugged straight into a bigger 2 switch Core and forget about ToR?

Do we need to think about any overlay networks that our compute minded colleagues might need to achieve some layer 2 stretchiness either between racks or between premises? 

Are there security appliances (real or virtual) and or load balancing appliances (again real or virtual) that we need to factor into our design?

Well,

Firstly, I'd look at 5700 in IRF pairs in the racks with a pair of 5940 sat over them. 40Gb Bidi conenctions to keep the cost down - loads of active-active 40Gb non-blocking. Think about including a segregated 1Gb network (can just be a single switch) just to plug the management ports into and keep them off the "prod" LAN.

The (formerly procurve) aruba-OS switches are great in the Campus but IMHO Comware* having IRF, mBGP, VRF capability, finegrained-QoS, storage networking, eVPN, VXLAN VTEP, QinQ, deep debug and front / back airflow makes them the clear winner in a datacentre / server connecting type role. 

* - Features vary by model etc :-) talking at least 5700/59x0 

Paint us a picture, give us a use case and we'll help best we can.

I am a fan of getting the footprint as small / lean as possible (max perf per watt or max perf per 1U of rack) and then using the least number of biggest bandwidth per $ links to lash it together. You might be able to do it all with 2 good switches :-) 

Hope that was useful - plenty to get stuck into.

Ian

Hi Ian,

Here is more info.. hope it helps.

I'll try to be as specific as possible, without going out of the subject.

Main Idea: 

• 3 racks (initially), with 1 for staging (not important) and 2 for production.
• Everything needs to be redundant, so if one racks goes down, we are still online.
• We have two internet lines dropped from our ISP (active/passive with bgp). One line goes to Rack #1, the other to Rack #2.

Each production racks (two) will have:

• Firewall (active/passive with heartbeat)
• load balalncer
• virtualisation servers

Future: Should be easy to grow horizontally at the virtualisation level. Future racks will have a ToR and servers. 

Our current idea:

• ToR on each rack (5700 or 3800 - depending on $$), 2 switches stacked, with LACP on servers, connected with 1Gb network
• Each ToR will uplink (SFP+) to our core/aggregation (one or two) switch(es) (5406R?) 
• Firewall will also go directly to core/aggregation
• Dropped line from ISP will also go directly to core/aggregation

Questions:

• Does this design make sense?
• Will this give us redundancy at all (network) levels?
• Should we get one or two core/aggregation switches?

Notes:

• We are not expecting a huge amount of data in the beginning, so 40Gb seems a bit too much.
• Traffic will come to the firewall which will then pass most of it to the load balancer.
• Firewall will also be the default gw of all servers (on all vlans)
• We will virtualise, but not use any virtual switch or so. Only Vlan tags on the server side.
• This network will participate in other networks, but via VPN tunnel (not sure if relevant)
• We are looking at more o less 10 physical servers per rack
• Simple is better. I prefer a easy to manage setup, since we will be dealing with it, and none of us is a network expert :)
• Price do matter, but also quality. (so, not too expensive and not the cheapest)
• Storage is not included here because it will have a separate network layer.

Thanks for the help!

Hi emcs,

@just a clarification, you wrote:

---

@emsc wrote:

Main Idea: 

• 3 racks (initially), with 1 for staging (not important) and 2 for production.
• Everything needs to be redundant, so if one racks goes down, we are still online.
• We have two internet lines dropped from our ISP (active/passive with bgp). One line goes to Rack #1, the other to Rack #2.

Each production racks (two) will have:

• Firewall (active/passive with heartbeat)
• load balalncer
• virtualisation servers

Our current idea:

• ToR on each rack (5700 or 3800 - depending on $$), 2 switches stacked, with LACP on servers, connected with 1Gb network
• Each ToR will uplink (SFP+) to our core/aggregation (one or two) switch(es) (5406R?) 
• Firewall will also go directly to core/aggregation
• Dropped line from ISP will also go directly to core/aggregation

---

Since you wrote "Each production racks (two) will have:"...does this mean that you're going to have one "HA Firewalls cluster" and Load Balancer per Rack (so two "HA Firewall Clusters" and two Load Balancers in total) or that you're going to have the Active Firewall on Production Rack 1 and the Passive Firewall on Production Rack 2 and just a single Load Balancer shared for both Racks?

If the right answer is a Go for the first case (Two "HA Firewall clusters" and two Load Balancers in total) this means that you're going to have two Active+Passive ISP Links per each Production Rack...is that right?

One answer or another, I imagine the (or each) HA Firewall Cluster (one unit Active, the other Passive/Standby) is going to be depolyed using VRRP, so all the network's hosts will be connected to the (virtual) Default Gateway IP Address provided by that HA Firewall Cluster.

If so your ISP will drop an Active and a Backup/Passive connectivity (2 links) on each Production Rack for each HA Firewall Cluster...so you're going to have a grand total of two Active and two Passive. Strange but totally possible (will be then interesting to see how you manage the second (virtual) Default Gateway IP Address provided by the second HA Firewall Cluster with respect to your Network hosts...something like a secondary/alternative virtual Default Gateway if the first isn't available to manage outgoing traffic).

Possibly here I'm totally wrong but all depends by that first sentence "Each production racks (two) will have:".

Clarification closed.

From what you wrote it seems that all edge connectivity (physical/virtual Default Gateway(s)) is going to be connected to the Distribution layer...and not to the Core one. If so you need to provide redundancy to the Distribution Layer...and - considering that Distribution layer will have clients only - the best is to go with Aruba 5406R deployed with dual Management Module and Redundant Power Supplies, if you are for the ArubaOS Switch on Distribution side.

It's interesting to note that, IMHO, you're doing the contrary of one would expect (it's hard to know if your important traffic should be the one destined to Servers or the one that is concurrently destined to Servers and also coming from Clients): the pyramid up-side-down, where the distribution layer has less units (at worst one, at best two in VSF) than the core (where you will have two ToR Switches deployed in IRF if Comware based or deployed with Hardware Stacking, if ProVsion based).

Again maybe I'm wrong and I totally misunderstood the whole picture...but it's what I'm able to figure reading your described setup.

Why not considering an all-Comware based setup that will span through both Racks deployed with a two full-Ring topologies (with MAD Device)...one ring for ToR and one for Distribution (with less expensive Switches than the HPE FlexNetwork 5700)?

Questions:

• Does this design make sense?
• Will this give us redundancy at all (network) levels?
• Should we get one or two core/aggregation switches?

In answer to the "redundancy at all (network) levels", I'd suggest consider having two dist/core switches. Here's why...

If you have a 5406R with redundant management modules for instance, there are some cases where downtime is required to perform maintenance functions, and there is only a single fan tray. There is only one software manager active, and even with all of the non-stop redundancy options configured properly, there are some cases where downtime can happen. For example software bugs, power failover issues (the power redundancy is not 100% uptime, power failures can cause small outages for components in the chassis).

I had a situation once where the fan controller in a 8206zl (older model, similar design) got wet, the fan went down, and the switch shutdown. In the 5406R there is more redundancy than the 8206zl, but still a single fan tray.

With two 5406R with VSF, 3810M Stacked Chassis, or Comware IRF at core/dist, you get a higher uptime due to every component being redundant, even the underlying hardware and operating system. Two physical systems in a virtual chassis will normally have higher uptime, but it just costs more. Chassis internal redundancy is great for the budget though.

I'd also be checking the guides and manuals for each model choice on how upgrades are performed, and what kind of downtime is present. At some point you will need downtime to make changes and upgrades. Some virtual switch designs have less downtime for management than others.

Btw, many DC fabric designs these days are moving away from traditional hierarchical designs and moving to spine and leaf mesh designs. The redundancy is built using TRILL/SPB or similar and the software is modular, or allows in service software upgrades (issu). These designs allow a higher uptime and may be a better match for a DC build. Sometimes DC designs are better served by flat mesh designs (flattened networks).

Hi Stephen,

Thanks for the info.. very useful!

There is a budget issue, so I thought to start with one 5406 (with everything redundant) and then, eventually, getting a second one.

Also, these 5406 will be both distribution (or aggregation) and core switches (via vlan). I don't know if this is optimanl, but we can't afford 3 layers of switches for our setup.

Questions:

• Is adding a second one possible or do I need to 'vanilla' switches to configure IRF?
• Can IRF be done with crossover or should it always be done on the uplink switch?

Thanks!

Btw, I would recommend you re-read the previous posts from Parnassus carefully, and the links shown.

With a single 5406R with redundant modules and no VSF, you can use cheaper v2 modules (or old ones you already have from the 5400/8200 v2 range). That keeps the costs down at the start.

With VSF you will need to run the two 5406R's with v3 modules, that can get expensive and VSF is still fairly new and relatively early in the development/bug fixing cycle. Maybe you should consider DT (dt-trunk) instead as that will work with v2 modules and the software has been around for a while now.

If at all possible, test your approach in a lab setting before going live. The 5406R chassis itself is fairly cheap, it's the modules that are expensive. I'd recommend when you want to move to VSF to purchase a second 5406R chassis, and build a VSF with a subset of v3 modules, and test the single chassis to VSF conversion procedure as thoroughly as you can. Consider testing the Layer 2 and Layer 3 functionality of either DT or VSF so you understand the differences between them on traffic flows (layer 2 and layer 3 have frame forwarding differences).

I'm not sure I'd want to introduce any solution using DT or VSF unless I've tested it first in a lab, and thoroughly understand the OAM (Operations, Admin, Management) of the solution (backups, upgrades, changes, etc) before rolling it into production use.

Avoid putting POE modules in a core/aggregation switch. I often have to interrupt services in my site due to POE controller failure, as many of my core switches have inherited modules with POE from other roles, and POE controllers tend to fail more frequently than anything else. Everytime I swap them out, non-POE services have downtime. You'll get higher uptime without POE in those switch roles.

I just re-read the whole thread again. I agree with the previous posters, comware may be a better fit for a pure DC deployment than Procurve which is more aimed at Campus networking. Procurve is missing many current generation DC features. Unless of course, the DC size and location will be finite, and you are relying on load balancers to manage redundancy across DC's.

One question, what networking products do you and your other staff have experience in, and who will be managing the network. If you do choose to go Comware, it has excellent features for DC networking, but you need a different skill set than managing the Procurves.