Wired Intelligent Edge

I would like more clarification to your response.  I have a vsx pair of 8325 switches that I want to set up mirroring on for all of my MC-LAGS.  I have a security appliance that the mirrored traffic will be sent to.  For the destination, would I set up a single 10G interface going to the appliance on just the primary switch in the mirror session and then specify all of the MC-LAGs as the source interface?  Does the same mirror session and source/destinations need to be set up on the secondary switch as well?  If so, do I need to have a MC-LAG going to my appliance from the destination interface on each VSX paired switch? 

@wdubose Hi! I'm interested about that too, how to deal with a VSX deployment when an IDS is used (on a bare metal server) as the destination of mirrored traffic (e.g. from all VLAN as sources): Server link to VSX Primary only or a Server LACP (Bond) to both VSX members (thus through a VSX LAG)? 

I would like more clarification to your response.  I have a vsx pair of 8325 switches that I want to set up mirroring on for all of my MC-LAGS.  I have a security appliance that the mirrored traffic will be sent to.  For the destination, would I set up a single 10G interface going to the appliance on just the primary switch in the mirror session and then specify all of the MC-LAGs as the source interface?  Does the same mirror session and source/destinations need to be set up on the secondary switch as well?  If so, do I need to have a MC-LAG going to my appliance from the destination interface on each VSX paired switch? 

What I have observed is there is no sync across VSX of mirroring. Each physical box is independent. Frames received on one member won't necessarily be seen on the other member. MC-LAG is just a way to negotiate the LACP with the partner switch.

When mirroring for troubleshooting purposes, the mirror from each member of the MC-LAG had frames received on that link. I noted that some frames were seen on both outputs when source was a vlan and not a physical interface. I questioned Aruba Support and they said this is because the frames are shared on active-gateways rather than the mirror session is sync'd.

Therefore I can't see any situation where if you want all traffic, you can avoid having a mirror session on each member of VSX and therefore the IDS would need two physicals.

I would also be cautious about mirroring all the MC-LAGs. Clearly there is the opportunity for over subscription and therefore the IDS missing packets. Which is something that will lead to false positives or worse.

Finally, mirroring to CPU is good to get a flavor of what is on the link but in testing I found huge packet loss, even at 100Mb traffic flows. So if while developing your solution you use this to observe be cautious. The absesense of frames doesn't mean they weren't on the wire. Mirroring to an interface is very different.

Original Message:
Sent: Jul 10, 2023 07:31 AM
From: parnassus
Subject: Mirroring in VSX Topology

@wdubose Hi! I'm interested about that too, how to deal with a VSX deployment when an IDS is used (on a bare metal server) as the destination of mirrored traffic (e.g. from all VLAN as sources): Server link to VSX Primary only or a Server LACP (Bond) to both VSX members (thus through a VSX LAG)? 

Original Message:
Sent: Jul 09, 2023 02:36 PM
From: wdubose
Subject: Mirroring in VSX Topology

I would like more clarification to your response.  I have a vsx pair of 8325 switches that I want to set up mirroring on for all of my MC-LAGS.  I have a security appliance that the mirrored traffic will be sent to.  For the destination, would I set up a single 10G interface going to the appliance on just the primary switch in the mirror session and then specify all of the MC-LAGs as the source interface?  Does the same mirror session and source/destinations need to be set up on the secondary switch as well?  If so, do I need to have a MC-LAG going to my appliance from the destination interface on each VSX paired switch? 

Original Message:
Sent: Oct 26, 2022 03:03 AM
From: vincent.giles
Subject: Mirroring in VSX Topology

Same command than for lag.

source interface lag9 both

(assuming lag9 is your VSX multi-chassis LAG).
Original Message:
Sent: Oct 25, 2022 06:12 AM
From: SHAY ELIASI
Subject: Mirroring in VSX Topology

searching for the best-practise for configuring mirroring ports on mc-lags of both members.
cant find any reference for mc-lags in the documentation (vsx,monitoring,security..).

please reply with relevant reference.

single chassis lag example,
switch(config-mirror-4)# source interface lag1 both

regards,

I wonder, since I have two SFP+ interfaces available on my security appliance, if I could configure an MC-lag on my two vsx switches and a lag on the appliance.  Then specify that MC-lag as the destination and then all of my downstream MC-Lags as the source.

If my only option is to have a single 10G link to my security appliance from my primary core and I have 30+ 10G access layer MC-lags as the source, will this overload the single 10G link?

Original Message:
Sent: Jul 11, 2023 03:59 AM
From: IanNightingale
Subject: Mirroring in VSX Topology

What I have observed is there is no sync across VSX of mirroring. Each physical box is independent. Frames received on one member won't necessarily be seen on the other member. MC-LAG is just a way to negotiate the LACP with the partner switch.

When mirroring for troubleshooting purposes, the mirror from each member of the MC-LAG had frames received on that link. I noted that some frames were seen on both outputs when source was a vlan and not a physical interface. I questioned Aruba Support and they said this is because the frames are shared on active-gateways rather than the mirror session is sync'd.

Therefore I can't see any situation where if you want all traffic, you can avoid having a mirror session on each member of VSX and therefore the IDS would need two physicals.

I would also be cautious about mirroring all the MC-LAGs. Clearly there is the opportunity for over subscription and therefore the IDS missing packets. Which is something that will lead to false positives or worse.

Finally, mirroring to CPU is good to get a flavor of what is on the link but in testing I found huge packet loss, even at 100Mb traffic flows. So if while developing your solution you use this to observe be cautious. The absesense of frames doesn't mean they weren't on the wire. Mirroring to an interface is very different.

Original Message:
Sent: Jul 10, 2023 07:31 AM
From: parnassus
Subject: Mirroring in VSX Topology

@wdubose Hi! I'm interested about that too, how to deal with a VSX deployment when an IDS is used (on a bare metal server) as the destination of mirrored traffic (e.g. from all VLAN as sources): Server link to VSX Primary only or a Server LACP (Bond) to both VSX members (thus through a VSX LAG)? 

Original Message:
Sent: Jul 09, 2023 02:36 PM
From: wdubose
Subject: Mirroring in VSX Topology

I would like more clarification to your response.  I have a vsx pair of 8325 switches that I want to set up mirroring on for all of my MC-LAGS.  I have a security appliance that the mirrored traffic will be sent to.  For the destination, would I set up a single 10G interface going to the appliance on just the primary switch in the mirror session and then specify all of the MC-LAGs as the source interface?  Does the same mirror session and source/destinations need to be set up on the secondary switch as well?  If so, do I need to have a MC-LAG going to my appliance from the destination interface on each VSX paired switch? 

Original Message:
Sent: Oct 26, 2022 03:03 AM
From: vincent.giles
Subject: Mirroring in VSX Topology

Same command than for lag.

source interface lag9 both

(assuming lag9 is your VSX multi-chassis LAG).
Original Message:
Sent: Oct 25, 2022 06:12 AM
From: SHAY ELIASI
Subject: Mirroring in VSX Topology

searching for the best-practise for configuring mirroring ports on mc-lags of both members.
cant find any reference for mc-lags in the documentation (vsx,monitoring,security..).

please reply with relevant reference.

single chassis lag example,
switch(config-mirror-4)# source interface lag1 both

regards,

Curious on the outcome of your question.

I will be setting up MC-LAG mirroring on a pair of 8360s in VSX.

Original Message:
Sent: Jul 11, 2023 11:08 AM
From: wdubose
Subject: Mirroring in VSX Topology

I wonder, since I have two SFP+ interfaces available on my security appliance, if I could configure an MC-lag on my two vsx switches and a lag on the appliance.  Then specify that MC-lag as the destination and then all of my downstream MC-Lags as the source.

If my only option is to have a single 10G link to my security appliance from my primary core and I have 30+ 10G access layer MC-lags as the source, will this overload the single 10G link?

Original Message:
Sent: Jul 11, 2023 03:59 AM
From: IanNightingale
Subject: Mirroring in VSX Topology

What I have observed is there is no sync across VSX of mirroring. Each physical box is independent. Frames received on one member won't necessarily be seen on the other member. MC-LAG is just a way to negotiate the LACP with the partner switch.

When mirroring for troubleshooting purposes, the mirror from each member of the MC-LAG had frames received on that link. I noted that some frames were seen on both outputs when source was a vlan and not a physical interface. I questioned Aruba Support and they said this is because the frames are shared on active-gateways rather than the mirror session is sync'd.

Therefore I can't see any situation where if you want all traffic, you can avoid having a mirror session on each member of VSX and therefore the IDS would need two physicals.

I would also be cautious about mirroring all the MC-LAGs. Clearly there is the opportunity for over subscription and therefore the IDS missing packets. Which is something that will lead to false positives or worse.

Finally, mirroring to CPU is good to get a flavor of what is on the link but in testing I found huge packet loss, even at 100Mb traffic flows. So if while developing your solution you use this to observe be cautious. The absesense of frames doesn't mean they weren't on the wire. Mirroring to an interface is very different.

Original Message:
Sent: Jul 10, 2023 07:31 AM
From: parnassus
Subject: Mirroring in VSX Topology

@wdubose Hi! I'm interested about that too, how to deal with a VSX deployment when an IDS is used (on a bare metal server) as the destination of mirrored traffic (e.g. from all VLAN as sources): Server link to VSX Primary only or a Server LACP (Bond) to both VSX members (thus through a VSX LAG)? 

Original Message:
Sent: Jul 09, 2023 02:36 PM
From: wdubose
Subject: Mirroring in VSX Topology

I would like more clarification to your response.  I have a vsx pair of 8325 switches that I want to set up mirroring on for all of my MC-LAGS.  I have a security appliance that the mirrored traffic will be sent to.  For the destination, would I set up a single 10G interface going to the appliance on just the primary switch in the mirror session and then specify all of the MC-LAGs as the source interface?  Does the same mirror session and source/destinations need to be set up on the secondary switch as well?  If so, do I need to have a MC-LAG going to my appliance from the destination interface on each VSX paired switch? 

Original Message:
Sent: Oct 26, 2022 03:03 AM
From: vincent.giles
Subject: Mirroring in VSX Topology

Same command than for lag.

source interface lag9 both

(assuming lag9 is your VSX multi-chassis LAG).
Original Message:
Sent: Oct 25, 2022 06:12 AM
From: SHAY ELIASI
Subject: Mirroring in VSX Topology

searching for the best-practise for configuring mirroring ports on mc-lags of both members.
cant find any reference for mc-lags in the documentation (vsx,monitoring,security..).

please reply with relevant reference.

single chassis lag example,
switch(config-mirror-4)# source interface lag1 both

regards,

Great!

Thank you.

My setup will be a little simpler.  Planning on this:

8360Core1

mirror session 1
    destination interface 1/1/12
    source lag1 both
    source lag2 both
    enable

8360Core2

mirror session 2
    destination interface 1/1/12
    source lag1 both
    source lag2 both
    enable

I'm very interested in the outcome of your configuration. I have made a similar setup although my source is a VLAN. (Source vlan 200 both) and my destination is a MCLAG - effectively the same LAG as seen from both VSX members. 
It seems to work, but there are sessions that are never seen in my flowanalyszer on the destination. No packets are reported dropped anywhere, so I have a sneaking suspicion that flows that goes through one of the VSX members are not mirrored correctly to the destination MCLAG.
Should I drop the destination MCLAG and make that two individual interfaces like in your examples?

It would be really nice if Aruba would chime in here with some documentation on the supported way of creating these mirrors of MCLAGs and VLANs on VSX pairs.
Fx. Is a MCLAG suported as a destination?

Great!

Thank you.

My setup will be a little simpler.  Planning on this:

8360Core1

mirror session 1
    destination interface 1/1/12
    source lag1 both
    source lag2 both
    enable

8360Core2

mirror session 2
    destination interface 1/1/12
    source lag1 both
    source lag2 both
    enable

Not sure about MCLAG as a destination.  Mine seems to work fine configured as is.  I found very little on the subject from Aruba.  It was just a matter of experimenting until it worked.

I'm very interested in the outcome of your configuration. I have made a similar setup although my source is a VLAN. (Source vlan 200 both) and my destination is a MCLAG - effectively the same LAG as seen from both VSX members. 
It seems to work, but there are sessions that are never seen in my flowanalyszer on the destination. No packets are reported dropped anywhere, so I have a sneaking suspicion that flows that goes through one of the VSX members are not mirrored correctly to the destination MCLAG.
Should I drop the destination MCLAG and make that two individual interfaces like in your examples?

It would be really nice if Aruba would chime in here with some documentation on the supported way of creating these mirrors of MCLAGs and VLANs on VSX pairs.
Fx. Is a MCLAG suported as a destination?

Great!

Thank you.

My setup will be a little simpler.  Planning on this:

8360Core1

mirror session 1
    destination interface 1/1/12
    source lag1 both
    source lag2 both
    enable

8360Core2

mirror session 2
    destination interface 1/1/12
    source lag1 both
    source lag2 both
    enable

Hi! any issue on the Security Appliance side with your (absolutely reasonable) approach? I mean (re)asking what I asked you here months ago when you were still experimenting. We are in the process of designing something very similar (probably with 100G DACs to the Security Appliance given the core role of the involved VSX Cluster). I'm really curious, also understanding which SA approach you used (black box or open, HW used...HPE or other vendors). My idea is to design something around an HPE ProLiant DL Gen10 or Gen11 with appropriate CPU, storage and moreover NICs. I'm still curious to see what Aruba engineers could advise us...as best practice.

Not sure about MCLAG as a destination.  Mine seems to work fine configured as is.  I found very little on the subject from Aruba.  It was just a matter of experimenting until it worked.

I'm very interested in the outcome of your configuration. I have made a similar setup although my source is a VLAN. (Source vlan 200 both) and my destination is a MCLAG - effectively the same LAG as seen from both VSX members. 
It seems to work, but there are sessions that are never seen in my flowanalyszer on the destination. No packets are reported dropped anywhere, so I have a sneaking suspicion that flows that goes through one of the VSX members are not mirrored correctly to the destination MCLAG.
Should I drop the destination MCLAG and make that two individual interfaces like in your examples?

It would be really nice if Aruba would chime in here with some documentation on the supported way of creating these mirrors of MCLAGs and VLANs on VSX pairs.
Fx. Is a MCLAG suported as a destination?

Great!

Thank you.

My setup will be a little simpler.  Planning on this:

8360Core1

mirror session 1
    destination interface 1/1/12
    source lag1 both
    source lag2 both
    enable

8360Core2

mirror session 2
    destination interface 1/1/12
    source lag1 both
    source lag2 both
    enable

@wdubose Hi! I'm interested about that too, how to deal with a VSX deployment when an IDS is used (on a bare metal server) as the destination of mirrored traffic (e.g. from all VLAN as sources): Server link to VSX Primary only or a Server LACP (Bond) to both VSX members (thus through a VSX LAG)? 

I would like more clarification to your response.  I have a vsx pair of 8325 switches that I want to set up mirroring on for all of my MC-LAGS.  I have a security appliance that the mirrored traffic will be sent to.  For the destination, would I set up a single 10G interface going to the appliance on just the primary switch in the mirror session and then specify all of the MC-LAGs as the source interface?  Does the same mirror session and source/destinations need to be set up on the secondary switch as well?  If so, do I need to have a MC-LAG going to my appliance from the destination interface on each VSX paired switch? 

Sorry to ressurrect this thread, but my mirroring is just not working as expected on my  VSX. I setteled on setting up a mirror session on each VSX node with the source being a VLAN interface and the destination being a dedicated port on each VSX node:

Regardless of what I do I only see a fraction of the packets being forwarded/recieved by the VLAN interface. But it's completely random. It rarely mirrors more than about 150mbit/s and when the source VLAN has very little activity most packets are actually forwarded. But since it usually is around 1 - 3 Gbit in utilisation the average of about 150mbit i get mirrored is only about 1/6th to 1/18th of the actual traffic being passed.

Is there some kind of limiter being used or does mirroring just not behave in a VSX pair even though i do it "individually" on each node?

OS version = 10.10.1090 on CX 8360

@wdubose Hi! I'm interested about that too, how to deal with a VSX deployment when an IDS is used (on a bare metal server) as the destination of mirrored traffic (e.g. from all VLAN as sources): Server link to VSX Primary only or a Server LACP (Bond) to both VSX members (thus through a VSX LAG)? 

I would like more clarification to your response.  I have a vsx pair of 8325 switches that I want to set up mirroring on for all of my MC-LAGS.  I have a security appliance that the mirrored traffic will be sent to.  For the destination, would I set up a single 10G interface going to the appliance on just the primary switch in the mirror session and then specify all of the MC-LAGs as the source interface?  Does the same mirror session and source/destinations need to be set up on the secondary switch as well?  If so, do I need to have a MC-LAG going to my appliance from the destination interface on each VSX paired switch? 

Sorry to ressurrect this thread, but my mirroring is just not working as expected on my  VSX. I setteled on setting up a mirror session on each VSX node with the source being a VLAN interface and the destination being a dedicated port on each VSX node:

Regardless of what I do I only see a fraction of the packets being forwarded/recieved by the VLAN interface. But it's completely random. It rarely mirrors more than about 150mbit/s and when the source VLAN has very little activity most packets are actually forwarded. But since it usually is around 1 - 3 Gbit in utilisation the average of about 150mbit i get mirrored is only about 1/6th to 1/18th of the actual traffic being passed.

Is there some kind of limiter being used or does mirroring just not behave in a VSX pair even though i do it "individually" on each node?

OS version = 10.10.1090 on CX 8360

@wdubose Hi! I'm interested about that too, how to deal with a VSX deployment when an IDS is used (on a bare metal server) as the destination of mirrored traffic (e.g. from all VLAN as sources): Server link to VSX Primary only or a Server LACP (Bond) to both VSX members (thus through a VSX LAG)? 

I would like more clarification to your response.  I have a vsx pair of 8325 switches that I want to set up mirroring on for all of my MC-LAGS.  I have a security appliance that the mirrored traffic will be sent to.  For the destination, would I set up a single 10G interface going to the appliance on just the primary switch in the mirror session and then specify all of the MC-LAGs as the source interface?  Does the same mirror session and source/destinations need to be set up on the secondary switch as well?  If so, do I need to have a MC-LAG going to my appliance from the destination interface on each VSX paired switch?