Wired Intelligent Edge

Switch model: Aruba JL084A ( 3810 M)

 ROM Version : KB.16.01.0009

Setup:

Firewall <--> Switch <-->WAN

TRK1 - fireawll -- switch

TRK 5 -- wan -- switch

From the WAN Router When I ping to the firewall...

TRK 5 -- all the packets received from the WAN Router but 8% packet loss while sending back to the WAN Router. 

TRK 1 -- All the packet going to the fireawll and all the packets received from the firewall.

8% packet are duplicated from firewall back to the switch and time stamp matches the packet which did not receives the reply on the TRK5 capture. 

Can someone please help me out to understand where in the switch packets are dropping. 

There is no TX or RX errors or incrementing errors on TRK1 or TRK5. 

Thank you,

Nilay.

Hi! Can you explain us why you used the TRK nomenclature for involved interfaces (downlink from Switch to Firewall, uplink from Switch to WAN Router)?

Are you really telling us you configured PORT AGGREGATION to both your Firewall and WAN Router?

Hi,

There is 2 HA firewalls with 4 x connections. 

on the WAN side there is only one connection but it was initially planned to have two HA Physical connection across nexus which didn't happen. 

Yes checked speed and duplex and it is looking fine. no drops on the interface or queues.

Thanks,

Nilay.

Image stamp: /ws/swbuildm/rel_washington_qaoff/code/build/bom(swbuildm_rel_washington_qaoff_rel_washington)
Jun 22 2018 12:34:17
KB.16.06.0006
923
Boot Image: Primary

Boot ROM Version: KB.16.01.0009
Active Boot ROM: Primary

Hi Nilay! I don't want to stress about that...but when you write:

---

@nilay wrote: There is 2 HA firewalls with 4 x connections. 

do you mean that the Firewall pair (HA Cluster) is seen as one logical entity?

Better rephrased: are you sure that you can aggregate 2 or 4 ports into a Port Trunk logical interface (say called "Trk1") on Switch side and then terminate all of its member physical links against any member of the Firewall HA Cluster without suffering from any issue since it is really seen as one single switch (as one single logical entity) and not as two standalone devices interconnected with just an heartbeat link?

If Firewall HA Cluster = one logical entity -> any physical port can be part of an aggregate interface (it can be done involving both Firewalls concurrently), those physical links can consequently terminate against the Switch (and vice-versa).

If Firewall HA Cluster = two logical entities -> any physical port within the very same Firewall appliance con be part of an aggregate interface valid for that very Firewall appliance (so each Firewall will have its aggregated logical interface), physical links of each aggregate interface will terminate against the Switch (the Switch , necessarily, will need two Port Trunk logical interfaces, say called "Trk1" for accepting links coming from FW-1 and "Trk2" for accepting links coming from FW-2). You can mix links coming from the aggregate interface defined on FW-1 into Trk2 and vice-versa.

It is the last one.

I have TRK1 in LACP with Firewall 1 and TRK2 in LACP in firewall 2.

It is check point firewall HA Cluster with active standby. 

Each firewall connects via two interfaces to both the stack switch in acitve active mode .. 

I have tried to remove the vlan which was causing the packet drop from the second firewall and everything started working. 

when I add the VLAN back on the secondary or standby firewall it is back to packet drops. 

but wireshark capture is showing all the reply from the primary firewall.

Not sure how VLAN extended to secondary firewall is causing the problem. 

I have even tried by failover the cluster but same result. 

---

@nilay wrote:

It is the last one.

 

I have TRK1 in LACP with Firewall 1 and TRK2 in LACP in firewall 2.

 

It is check point firewall HA Cluster with active standby. 

 

Each firewall connects via two interfaces to both the stack switch in acitve active mode .. 

You have a strange way to represent your scenario...with your statement: "Each firewall connects via two interfaces to both the stack switch in acitve active mode .." are you now meaning you have a stack of Aruba 3810M and not just one standalone Aruba 3810M? that's just to figure out WHAT is your real networking scenario.

It's interesting: this thread started with one Firewall and one Switch and now we are discussing about two Firewalls in HA mode and (probably) a stack of Aruba 3810M.

Back on track.

If:

• Firewall 1 LAG-1 (link 1+2) goes to Aruba 3810M (or to a stack of two of them, it doesn't matter)
• Firewall 2 LAG-1 (link 1+2) goes to Aruba 3810M (or to a stack of two of them, it doesn't matter)

I don't see any issue about that, it is a totally reasonable topology. Clearly involved LAGs should be correctly set on both ends and VLAN tagging should be properly set too to cope with what was set on the Firewall's LAGs (and both LAGs should be identically configured - Firewall 1 vs Firewall 2 - because we're speaking about an HA Cluster).

Can you provide some information about Port Trunks (Trk1, Trk2) and their VLAN memberships? and also some information about how LAGs were configured on Firewall HA side?

Hi,

Following are the conifugration

interface Trk1
tagged vlan 909,2505,3500-3501,3505-3508,3650-3651,3710-3711
untagged vlan 1
spanning-tree priority 4
exit

interface Trk2
tagged vlan 909,2505,3500-3501,3505-3508,3650-3651,3710-3711
untagged vlan 1
spanning-tree priority 4
exit

LACP Trunk Port LACP Admin Oper
Port Enabled Group Status Partner Status Key Key
----- ------- ------- ------- ------- ------- ------ ------
1/1 Active Trk1 Up No Success 0 962
1/2 Active Trk2 Up No Success 0 963

2/1 Active Trk1 Up No Success 0 962
2/2 Active Trk2 Up No Success 0 963

----------------------------

Firewall is configured 

ClusterXL -- Active/ Standby

Bond 1 : Active/Active Round robin.  on both the active and standby firewalls. 

---- 

Hi!

can you share the (sanitized but not abridged) output of these commands:

• show trunks
• show lacp
• show vlan port trk1 detail
• show vlan port trk2 detail

related to Trk1 and Trk2 (provided that Trk1 to Active Firewall, Trk2 to Backup Firewall).

On the Check Point ClusterXL HA cluster set the Bond Load Sharing operating mode of bonds linked to the Aruba stack as IEEE 802.3ad (LACP) and not Round-Robin.

 sh trunks

Load Balancing Method: L3-based (default)

Port | Name Type | Group Type
------ + -------------------------------- ---------- + ------ --------
1/1 | Firewall1-Eth1-01 SFP+SR | Trk1 LACP
1/2 | Firewall2-Eth1-02 SFP+SR | Trk2 LACP

1/14 | To_WAN SFP+LR | Trk5 LACP

2/1 | Firewall1-Eth1-01 SFP+SR | Trk1 LACP
2/2 | Firewall2-Eth1-02 SFP+SR | Trk2 LACP

2/14 | RESERVED AS BACKUP PORT WAN| Trk5 LACP

==============================================

 sh lacp

LACP

LACP Trunk Port LACP Admin Oper
Port Enabled Group Status Partner Status Key Key
----- ------- ------- ------- ------- ------- ------ ------
1/1 Active Trk1 Up No Success 0 962
1/2 Active Trk2 Up No Success 0 963

1/14 Active Trk5 Up Yes Success 0 966

2/1 Active Trk1 Up No Success 0 962
2/2 Active Trk2 Up No Success 0 963

2/14 Active Trk5 Down No Success 0 966
=============================================

 sh vlan port trk1 detail

Status and Counters - VLAN Information - for ports Trk1

VLAN ID Name | Status Voice Jumbo Mode
------- -------------------- + ---------- ----- ----- --------
1 DEFAULT_VLAN | Port-based No No Untagged
909 WEX | Port-based No No Tagged
2505 SSS-TransitVlan | Port-based No No Tagged
3500 MPLS-TransitVlan | Port-based No No Tagged
3501 Management | Port-based No No Tagged
3650 Wireless Guest | Port-based No No Tagged
3651 Wireless BYOD | Port-based No No Tagged
3710 WAN-Peering_... | Port-based No No Tagged
3711 InterDC_Peering_1... | Port-based No No Tagged

 sh vlan port trk2 detail

Status and Counters - VLAN Information - for ports Trk2

VLAN ID Name | Status Voice Jumbo Mode
------- -------------------- + ---------- ----- ----- --------
1 DEFAULT_VLAN | Port-based No No Untagged
909 WEX | Port-based No No Tagged
2505 SSS-TransitVlan | Port-based No No Tagged
3500 MPLS-TransitVlan | Port-based No No Tagged
3501 Management | Port-based No No Tagged
3650 Wireless Guest | Port-based No No Tagged
3651 Wireless BYOD | Port-based No No Tagged
3710 WAN-Peering_... | Port-based No No Tagged
3711 InterDC_Peering_1... | Port-based No No Tagged

Found the solution. It is realted to checkpoint issue. 

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk34180

Secondary/Standby checkpoint is seding DNS request using VIP as source IP address. When reply came from DNS server it go to Primary checkpoint (which as VIP). Primary checkpoint is aware of the original request so it Switch the packe to secondary checkpoint. 

As check point suggest to connect primarya nd secondary via switch, when packet switched from Primary to Secondary it dose nto change the source Mac address which is WAN router. 

Switch see the packet came from the Primary checkpoint with the WAN router mac address and it updated it's mac table which cause the flaps. 

It is resolved by no nat rule in the checkpoint to make sure check point dose not NAT to VIP address when traffic is originted from checkpoint it self. 

Do you have check the speed (and duplex) on router interface ?