I'm evaluating using Aruba CX 8360 for routing with a huge number of bgp routes. From the specs it should handle about 600k routes but that seems not to be unachievable.
The switch has learned less than 300k routes (ipv4+ipv6 combined):
router# show ip route summary
IPv4 Route Table Summary
VRF name : default
Protocol Active Routes
------------- -------------
connected 6
local 7
ospfv2 324
bgp 3166
router# show ipv6 route summary
IPv6 Route Table Summary
VRF name : default
Protocol Active Routes
------------- -------------
connected 6
local 7
ospfv3 73
bgp 125421
And the capabilities seems to be fine:
rtr-c1-dcg1# show capacities-status l3-resources
System Capacities Status: Filter L3 Resources
Capacities Status Name Value Maximum
-----------------------------------------------------------------------------------------
Number of IP neighbor (IPv4+IPv6) entries 162 65536
Number of IP Directed Broadcast neighbor entries 0 1024
Number of IPv4 neighbor(ARP) entries 141 65536
Number of IPv6 neighbor(ND) entries 21 65536
Number of L3 Groups for IP Tunnels and ECMP Groups currently configured 38 2000
Number of L3 Destinations for Routes, Nexthops in ECMP groups and Tunnels currently configured 33 4093
Number of routes (IPv4+IPv6) currently configured 128880 631290
Number of IPv4 routes currently configured 3404 630780
Number of IPv6 routes currently configured with prefix 0-64 125473 598014
Number of IPv6 routes currently configured with prefix 65-127 3 510
After observing black holing through the router it seems that not all routes are getting into the FIB. I finally was able to find it in the syslog (/var/log/messages) a huge number of logging messages for a random selection of prefixes:
2024-07-05T11:45:32.923337+00:00 router switchd_agent[3748]: debug|LOG_ERR|AMM|-|L3|L3_ASIC|Hardware Route, create failed for prefix: 2001:1A40:15FE::/47 vrf: 1 dest_id: 3 dest_fwd_type: route_ecmp_member dp_state: SINGLE due to OUT_OF_ROUTE. Total err_count=293514
Restarting the bgp sessions will trigger those messages again. It looks like if the FIB does not get the valid routes of the RIB and bricks the routing:
The hpe-routing daemon is running inside of the netns swns and I assume it is not aware that the route is missing in the FIB. Other routers might use the switch as next-hop based on a routing protocol, but the switch will not be able to route the packet correctly (according to the RIB) if the route is missing in the FIB 🤯🤯🤯
Any ideas?