Wired Intelligent Edge

 View Only
last person joined: yesterday 

Bring performance and reliability to your network with the Aruba Core, Aggregation, and Access layer switches. Discuss the latest features and functionality of the ArubaOS-Switch and ArubaOS-CX devices, and find ways to improve security across your network to bring together a mobile first solution.
Expand all | Collapse all

BGP default route flapping in VSX pair

Jump to Best Answer
This thread has been viewed 39 times
  • 1.  BGP default route flapping in VSX pair

    Posted Jan 08, 2022 05:58 AM
    We have 2x 6400 connected with iBGP routing, peerin withing VRFs (vrf lite type of thing). If I shutdown the first switches uplink VLAN I get default route from switch 2, then it goes away, comes back to routing table, goes away etc. After a while flapping it settles and is left in the routing table.

    Anyone seen anything similar or any ideas what might cause this?

    Both 6400's are connected to different core routers with eBGP. External routes advertise only 0.0.0.0/0 to this VRF. While it is flapping the route is in BGP routing table just fine


  • 2.  RE: BGP default route flapping in VSX pair

    Posted Jan 10, 2022 02:21 AM
    Hi

    can you please share your BGP config of the two 6400 router and some output like "show ip route", "show bgp all summary" and "show bgp all neighbor x.x.x.x advertised/received" ? One typical source of such a problem is the missing "next-hop-self" option on the iBGP peering between the two 6400. A BGP route does only get installed to the FIB of the nexthop is reachable to the router. So if you are relying on the other router's default prefix from the eBGP peering the nexthop would be the external router's address which may not be reachable by the not-directly-connected router. For this reason one would typically use the own address to advertise the external route internally using iBGP.

    Best,
    Thomas

    ------------------------------
    Thomas Siegenthaler
    ------------------------------



  • 3.  RE: BGP default route flapping in VSX pair

    Posted Jan 10, 2022 02:45 AM
    Hi,

    this is after I've shut down the CX-1 - core link. After shutting it down and doing show ip ro vrf vrf1 I can see default there, then it's gone and back again etc. I did forget to add next-hop self, but even after adding it I'm getting the same behaviour

    CX-1:
    router bgp 64921
    bgp router-id 10.100.6.102
    vrf vrf1
    neighbor 10.100.29.192 remote-as 64775
    neighbor 10.100.29.192 description external
    neighbor 10.100.29.192 timers 3 9
    neighbor 10.100.54.19 remote-as 64921
    neighbor 10.100.54.19 description cx2
    neighbor 10.100.54.19 fall-over bfd
    address-family ipv4 unicast
    neighbor 10.100.29.192 activate
    neighbor 10.100.29.192 advertisement-interval 0
    neighbor 10.100.54.19 activate
    neighbor 10.100.54.19 next-hop-self
    redistribute connected

    CX-1# show ip ro vrf vrf1
    0.0.0.0/0, vrf vrf1
    via 10.100.54.19, [200/0], bgp

    Address-family : IPv4 Unicast
    -----------------------------
    Neighbor Remote-AS MsgRcvd MsgSent Up/Down Time State AdminStatus
    10.100.29.192 64775 263346 318284 00h:04m:41s Idle Up
    10.100.54.19 64921 40692 47551 00h:07m:09s Established Up

    interface vlan3962
    vrf attach vrf1
    ip address 10.100.54.18/31

    CX-11# show bgp vrf vrf1 ipv4 unicast neighbors 10.100.54.19 routes
    Network Nexthop Metric LocPrf Weight Path
    *>i 0.0.0.0/0 10.100.54.19 0 100 0 64775 65300 65301 65442 ?

    CX-2:
    router bgp 64921
    bgp router-id 10.100.6.104
    vrf vrf1
    neighbor 10.100.54.16 remote-as 64775
    neighbor 10.100.54.16 description external
    neighbor 10.100.54.16 timers 3 9
    neighbor 10.100.54.18 remote-as 64921
    neighbor 10.100.54.18 description internal
    neighbor 10.100.54.18 fall-over bfd
    address-family ipv4 unicast
    neighbor 10.100.54.16 activate
    neighbor 10.100.54.16 advertisement-interval 0
    neighbor 10.100.54.16 route-map ADD-COST-100 out
    neighbor 10.100.54.18 activate
    neighbor 10.100.54.18 next-hop-self
    neighbor 10.100.54.18 route-map LOCAL-PREF-150 in
    redistribute connected

    CX-2# show ip ro vrf vrf1
    0.0.0.0/0, vrf vrf1
    via 10.100.54.16, [20/0], bgp

    CX-2# show bgp vrf vrf1 ipv4 unicast neighbors 10.100.54.18 adv
    Network Nexthop Metric LocPrf Weight Path
    *>i 0.0.0.0/0 10.100.54.19 0 100 0 64775 65300 65301 65442 ?



  • 4.  RE: BGP default route flapping in VSX pair

    EMPLOYEE
    Posted Jan 10, 2022 04:45 AM
    Is 10.100.54.18 a "connected" IP on CX-2 ?
    Did you perform a clear bgp * after adding next-hop-self ?

    ------------------------------
    Vincent Giles
    ------------------------------



  • 5.  RE: BGP default route flapping in VSX pair

    Posted Jan 10, 2022 04:49 AM
    Yes 10.100.54.18/31 is configured on VLAN interface and that VLAN is allowed between CX switches. I did clear the BGP session, and now CX1 can see the next hop for 0.0.0.0/0 as 10.100.54.19 (I mean I did it before sending those configs to this discussion, not now)


  • 6.  RE: BGP default route flapping in VSX pair

    Posted Jan 10, 2022 06:53 AM
    To me it seems as everything is as it should be (config-wise). Maybe one thing to add is "soft-reconfiguration inbound" option in order to not be dependent on clearing the BGP session everytime you change policies/config.

    If the issue still persists, I would now start doing some debugging, making sure the BGP session is stable (e.g. no constant UPDATE messages arriving from the peer revoking and advertising the default route). BGP debugging may lead you to the reason why the route is constantly installed and revoked again.
    Did you try to disable BFD for a second to check whether the issue persists? And am I correct that the two 6400 are the only routers in AS 64921 (--> iBGP requires a full-mesh or a route-reflector design to work properly)?
    What software version are you using? Did you check the release notes whether there is any bugs fixed in newer versions?

    Best,
    Thomas

    ------------------------------
    Thomas Siegenthaler
    ------------------------------



  • 7.  RE: BGP default route flapping in VSX pair

    Posted Jan 10, 2022 07:23 AM
    I did debug routeresolution and  there are messages like these:
    2022-01-10:09:52:02.263716|fibapp|LOG_DEBUG|AMM|1/1|ROUTERESOLUTION|ROUTERESOLUTION|RR Core: rr_idl_notify_core
    2022-01-10:09:52:02.264135|fibapp|LOG_DEBUG|AMM|1/1|ROUTERESOLUTION|ROUTERESOLUTION|RR Core received a notification for ROUTE component
    2022-01-10:09:52:02.264536|fibapp|LOG_DEBUG|AMM|1/1|ROUTERESOLUTION|ROUTERESOLUTION|RR Core: Route Delete
    2022-01-10:09:52:02.264921|fibapp|LOG_DEBUG|AMM|1/1|ROUTERESOLUTION|ROUTERESOLUTION_IPV4_ROUTE|RR Core: Route Data received: Prefix 0.0.0.0/0, Addr_family ipv4, Vrf-ID 5, Event DEL
    2022-01-10:09:52:02.265652|fibapp|LOG_DEBUG|AMM|1/1|ROUTERESOLUTION|ROUTERESOLUTION_IPV4_ROUTE|Route successfully deleted into the local cache table
    2022-01-10:09:52:02.278368|fibapp|LOG_DEBUG|AMM|1/1|ROUTERESOLUTION|ROUTERESOLUTION|RR Core: Route ADD
    2022-01-10:09:52:02.279033|fibapp|LOG_DEBUG|AMM|1/1|ROUTERESOLUTION|ROUTERESOLUTION_IPV4_ROUTE|RR Core: Route Data received: Prefix 0.0.0.0/0, Addr_family ipv4, Vrf-ID 5, Event ADD
    2022-01-10:09:52:02.279444|fibapp|LOG_DEBUG|AMM|1/1|ROUTERESOLUTION|ROUTERESOLUTION_IPV4_ROUTE|Route successfully added into the local cache table
    2022-01-10:09:52:02.279880|fibapp|LOG_DEBUG|AMM|1/1|ROUTERESOLUTION|ROUTERESOLUTION|Route Done Message: Route Event DONE, routes count 0



    Not sure though what that means... default route is in the BGP routing table all the time it is flapping in the IPv4 routing table. 

    These two switches are only ones in 64921. They're running an older version of software but currently we're too afraid to upgrade until we get VSX pair failover to work properly. We've had lot's of issues with UBT clients ending up in failed state after every small glitch in the network and requiring lot's of manual work to reauthenticate them.




  • 8.  RE: BGP default route flapping in VSX pair

    EMPLOYEE
    Posted Jan 10, 2022 08:06 AM
    Next step is to share topology and config, show ip route, show ip ospf, sh bgp nei.
    or contact support,

    ------------------------------
    Vincent Giles
    ------------------------------



  • 9.  RE: BGP default route flapping in VSX pair

    Posted Jan 10, 2022 08:15 AM
    show ip route was there for the relevant default-route part already in the paste. There's no OSPF running, neighbor status was also in the paste (CX wouldn't show advertised-routes either if the BGP session was down). I've also opened a TAC case but it'll probably take quite a while to get something from there.


  • 10.  RE: BGP default route flapping in VSX pair
    Best Answer

    Posted Jan 13, 2022 12:25 PM
    I added bgp fast-external-fallover and configured BFD, after that I can do failover and lose 1-2 pings from a linux server. I think it's as good as it's going to be as it seems that 500ms is the fastest rx/tx BFD values supported.