Hello,
I've seen some discussions already about the subject but so far, I could not find the answer to make this work on our environment.
Basically, we have a site where we need to ping the access points from one of our centralized platforms ( I know, ICMP is not ideal, but it is what it is...), and we can only do it for a short period, while the APs are booting.
This is clearly because of CPSec M.O, as we can see on the outputs below, so we will assymetry between the paths of the echo-request and echo-reply, but what is actually happening is that the traffic is being dropped by the controller (if I'm seeing this right). Now, as suggested in other discussions, I've made sure that ip routing in the controller is active, and also that we are not doing stateful ICMP processing.
From the information that we were able to gather so far, The datapath session table entry for the echo-reply is flagged with the D(rop), but honestly I'm yet to figure out the reason for that.
Strangely enough, we have another site, which is working as we intended. So on that site we can actually ping the APs from the same platform. We have compared the setups and, everything seems the same, in terms of firewall parameters and profiles,etc.
The working site is running 8.5.0.12 at the moment and we did find something in the
Resolve issues section that might be related:
Here's the output from the
faulty site:
(During the tests, we had a continuous ping to 192.168.160.148)
tcpdump from the site gateway, where we don't see the return traffic (we do some DNAT in here, but not related to the issue)
site-gateway:~# tcpdump -i any host 192.168.160.148 or host 10.166.65.148
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
09:08:35.307124 IP 172.16.8.126 > 10.166.65.148: ICMP echo request, id 1, seq 63610, length 40
09:08:35.307178 IP 172.16.8.126 > 192.168.160.148: ICMP echo request, id 1, seq 63610, length 40
09:08:35.307185 ethertype IPv4, IP 172.16.8.126 > 192.168.160.148: ICMP echo request, id 1, seq 63610, length 40
09:08:40.208347 IP 172.16.8.126 > 10.166.65.148: ICMP echo request, id 1, seq 63611, length 40
09:08:40.208393 IP 172.16.8.126 > 192.168.160.148: ICMP echo request, id 1, seq 63611, length 40
09:08:40.208400 ethertype IPv4, IP 172.16.8.126 > 192.168.160.148: ICMP echo request, id 1, seq 63611, length 40
09:08:40.498943 ARP, Request who-has 192.168.160.148 tell eagle-one.eagle-shs.com, length 28
09:08:40.498953 ethertype ARP, ARP, Request who-has 192.168.160.148 tell eagle-one.eagle-shs.com, length 28
09:08:40.499135 ethertype ARP, ARP, Reply 192.168.160.148 is-at 70:3a:0e:c4:41:34 (oui Unknown), length 46
09:08:40.499135 ARP, Reply 192.168.160.148 is-at 70:3a:0e:c4:41:34 (oui Unknown), length 46
09:08:45.212772 IP 172.16.8.126 > 10.166.65.148: ICMP echo request, id 1, seq 63612, length 40
09:08:45.212815 IP 172.16.8.126 > 192.168.160.148: ICMP echo request, id 1, seq 63612, length 40
09:08:45.212822 ethertype IPv4, IP 172.16.8.126 > 192.168.160.148: ICMP echo request, id 1, seq 63612, length 40
09:08:50.202495 IP 172.16.8.126 > 10.166.65.148: ICMP echo request, id 1, seq 63613, length 40
09:08:50.202536 IP 172.16.8.126 > 192.168.160.148: ICMP echo request, id 1, seq 63613, length 40
09:08:50.202543 ethertype IPv4, IP 172.16.8.126 > 192.168.160.148: ICMP echo request, id 1, seq 63613, length 40
Checked datapath of the AP
(RUHMDWLC01_B01_MDF) *[mynode] #show datapath session ip-addr 192.168.160.148 | include 172.16.8.126
Datapath Session Table Entries
------------------------------
Flags: F - fast age, S - src NAT, N - dest NAT
D - deny, R - redirect, Y - no syn
H - high prio, P - set prio, T - set ToS
C - client, M - mirror, V - VOIP
Q - Real-Time Quality analysis
u - Upstream Real-Time Quality analysis
I - Deep inspect, U - Locally destined
E - Media Deep Inspect, G - media signal
r - Route Nexthop, h - High Value
A - Application Firewall Inspect
B - Permanent, O - Openflow
L - Log, o - Openflow config revision mismatched
AP Flags: 1 - Class 1, 2 - Class 2, 3 - Class 3, w - In hardware
Source IP or MAC Destination IP Prot SPort DPort Cntr Prio ToS Age Destination TAge Packets Bytes Flags AP Flags CPU ID
----------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- ---------- ---------- --------------- --------------- -------
192.168.160.148 172.16.8.126 1 64057 0 0 0 0 1 dev4 69 -- -- FYI 0
192.168.160.148 172.16.8.126 1 64059 0 0 0 0 0 dev4 37 -- -- FYI 0
192.168.160.148 172.16.8.126 1 64058 0 0 0 0 1 dev4 51 -- -- FYI 0
192.168.160.148 172.16.8.126 1 64061 0 0 0 0 0 dev4 5 -- -- FYI 0
192.168.160.148 172.16.8.126 1 64060 0 0 0 0 0 dev4 1e -- -- FYI 0
172.16.8.126 192.168.160.148 1 64061 2048 0 0 0 0 dev4 5 -- -- FYCI 0
172.16.8.126 192.168.160.148 1 64060 2048 0 0 0 1 dev4 1e -- -- FYCI 0
172.16.8.126 192.168.160.148 1 64057 2048 0 0 0 1 dev4 69 -- -- FYCI 0
172.16.8.126 192.168.160.148 1 64059 2048 0 0 0 1 dev4 37 -- -- FYCI 0
172.16.8.126 192.168.160.148 1 64058 2048 0 0 0 1 dev4 51 -- -- FYCI 0
(RUHMDWLC01_B01_MDF) *[mynode] #
checked datapath session table on the controller
(Aruba-controller) *[mynode] #show datapath session table | include 172.16.8.126
172.16.8.126 192.168.160.2 6 64999 22 1/15784 0 0 0 0/0/0 42 112 10013 C 11
192.168.160.2 172.16.8.126 6 4343 65015 0/0 0 0 0 0/0/0 14 9 1009 F 11
192.168.160.2 172.16.8.126 6 4343 65014 0/0 0 0 0 0/0/0 14 9 1421 F 11
192.168.160.148 172.16.8.126 1 4454 0 0/0 0 0 0 tunnel 665 1 0 0 FDYC 13
192.168.160.148 172.16.8.126 1 4453 0 0/0 0 0 0 tunnel 665 6 0 0 FDYC 13
Here's the firewall parameters
(Aruba-controller) ^*[mynode] #show firewall
Global firewall policies
------------------------
Policy Action Rate Port
------ ------ ---- ----
Enforce TCP handshake before allowing data Disabled
Prohibit RST replay attack Disabled
Deny all IP fragments Disabled
Prohibit IP Spoofing Enabled
Monitor ping attack Disabled
Monitor TCP SYN attack Disabled
Monitor IP sessions attack Disabled
Deny inter user bridging Enabled
Log all received ICMP errors Disabled
Per-packet logging Disabled
Blacklist Grat ARP attack client Disabled
Allow tri-session with DNAT Disabled
Disable FTP server No
Blacklist ARP attack client Disabled
Monitor ARP attack Disabled
Monitor Gratuitous ARP attack Enabled 50/30sec
GRE call id processing Disabled
Session Idle Timeout Enabled 16 sec
WMM content enforcement Disabled
Trust packet QoS Disabled
Only allow local subnets in user table Disabled
Monitor/police CP attacks Disabled
Rate limit CP untrusted ucast traffic Enabled 9765 pps
Rate limit CP untrusted mcast traffic Enabled 3906 pps
Rate limit CP trusted ucast traffic Enabled 65535 pps
Rate limit CP trusted mcast traffic Enabled 3906 pps
Rate limit CP route traffic Enabled 976 pps
Rate limit CP session mirror traffic Enabled 976 pps
Rate limit CP auth process traffic Enabled 976 pps
Rate limit CP vrrp traffic Enabled 512 pps
Rate limit CP ARP traffic Enabled 3906 pps
Rate limit CP L2 protocol/other traffic Enabled 1953 pps
Deny inter user traffic Enabled
Prohibit ARP Spoofing Disabled
Enforce bw contracts for broadcast traffic Disabled
Multicast automatic shaping Disabled
Stall Detection Enabled
Enforce TCP Sequence numbers Disabled
AMSDU Rx Enabled
Jumbo Frames Disabled
Session-tunnel FIB Enabled
Prevent DHCP exhaustion Disabled
Deny source routing Disabled
Immediate Freeback Disabled
Stateful ICMP Processing Disabled
Optimize Duplicate Address Detection frames Enabled
Mcast RED Disabled
IPSec Mark Management Frames Disabled
Rate limit CP IKE traffic Disabled
Wireless Bridge Aging Enabled
Port Packet Drop Log Enable Disabled
App performance monitoring Disabled
DHCP performance monitoring Disabled
Drop Larger than GRE MTU DF frame, send ICMP Err Disabled
Drop Larger than GRE MTU DF frame Disabled
Drop Larger than GRE MTU frame, send ICMP Err Disabled
Drop Larger than GRE MTU frame Disabled
Enable GRE Inner Frame Fragmentation Disabled
Track Spoofs in Data Path Disabled
Rate limit CP IP Error pkts Enabled 128 pps
How can we verify the reason for the drop?
Any help will be much appreciated!
Best regards
------------------------------
Ricardo Marques
------------------------------