Controller-less WLANs

When trying to access the IAP by SSH, there could be a delay of about 30 seconds some times. This easily makes someone thing that the IAP is responding very slow to the SSH access request. However, the actual problem would be the DNS server IP configured on the IAP that is unreachable. 

A linux SSH server would perform a reverse DNS lookup of the SSH client's IP address (for security purpose) on receiving an SSH access request. This is done by the Linux SSH servers by default. 

Since IAP is based on Linux, the SSH server in the IAP would also perform a reverse DNS lookup of the SSH client's IP address while receiving an SSH access request. 

If the DNS IP address configured on the IAP is not reachable, then the DNS lookup would timeout after 10 seconds. However the IAP would try the DNS lookup three times before granting SSH access. The three timeouts causes a delay of 30 seconds (3 x 10 seconds) in total, after which the SSH access is granted. Hence the password prompt appears delayed.  

This behavior can be confirmed by debugging the dns packet on the IAP where we can see the three reverse DNS lookup tries to the SSH client's IP address at an interval of 10 seconds each. To perform the dns debug, login to the IAP by SSH and enter the below commands, 

# debug pkt type dns
# debug pkt dump

Then open another SSH session to the IAP. While the second SSH session is getting delayed by 30 seconds, we can see the reverse DNS lookup attempts on the first debug SSH window. Below is an example DNS debug output during an SSH delay. Client IP is 10.20.25.37. DNS server IP configured on the IAP is 192.168.192.168 that is unreachable, hence the DNS request would timeout. Note the time stamp interval (10 seconds) of the reverse DNS lookup. The reverse DNS lookup can be identified by the domain name ending with ".in-addr.arpa" and the client IP in reverse order. 

Rajaguru-IAP-109# debug pkt type dns
Rajaguru-IAP-109# debug pkt dump
If packet is of type DNS 
Press 'q' to quit.

Received packet from br0 (timestamp (1970-0-1 08:57:01:292998) )
[asap_firewall_forward(5048):firewall entry] len 84, vlan 0, egress CP, ingress br0:
  #mac: etype 0800 smac 6c:f3:7f:c5:2e:dc dmac 00:0b:86:86:09:80
  #ip: sip 10.17.170.26, dip 192.168.192.168, proto 17, dscp 0, dont fragment, last fragment, fragment offset 0
    #udp: sport 49523 dport 53 len 50
      #dns: message-type: standard query, txn id: 2
             domain name: 37.25.20.10.in-addr.arpa, type: PTR(12), class: IN(1)
[asap_firewall_forward(5218):vlan decision] len 84, vlan 1, egress CP, ingress br0:
[asap_firewall_forward(5894):bridge section] len 84, vlan 1, egress CP, ingress br0:
[asap_firewall_forward(6023):session section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6217):fastpath returned 1 opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6249):slowpath section: opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6485):back to fastpath, opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6023):session section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6217):fastpath returned 1 opcode 7] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6249):slowpath section: opcode 7] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6485):back to fastpath, opcode 3] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6789):route section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6839):cp route section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(7122):forward section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(7439):forwarding packet to bond0] len 84, vlan 1, egress bond0, ingress bond0:


Received packet from br0 (timestamp (1970-0-1 08:57:11:292849) )
[asap_firewall_forward(5048):firewall entry] len 84, vlan 0, egress CP, ingress br0:
  #mac: etype 0800 smac 6c:f3:7f:c5:2e:dc dmac 00:0b:86:86:09:80
  #ip: sip 10.17.170.26, dip 192.168.192.168, proto 17, dscp 0, dont fragment, last fragment, fragment offset 0
    #udp: sport 49526 dport 53 len 50
      #dns: message-type: standard query, txn id: 3
             domain name: 37.25.20.10.in-addr.arpa, type: PTR(12), class: IN(1)
[asap_firewall_forward(5218):vlan decision] len 84, vlan 1, egress CP, ingress br0:
[asap_firewall_forward(5894):bridge section] len 84, vlan 1, egress CP, ingress br0:
[asap_firewall_forward(6023):session section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6217):fastpath returned 1 opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6249):slowpath section: opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6485):back to fastpath, opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6023):session section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6217):fastpath returned 1 opcode 7] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6249):slowpath section: opcode 7] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6485):back to fastpath, opcode 3] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6789):route section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6839):cp route section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(7122):forward section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(7439):forwarding packet to bond0] len 84, vlan 1, egress bond0, ingress bond0:


Received packet from br0 (timestamp (1970-0-1 08:57:21:292849) )
[asap_firewall_forward(5048):firewall entry] len 84, vlan 0, egress CP, ingress br0:
  #mac: etype 0800 smac 6c:f3:7f:c5:2e:dc dmac 00:0b:86:86:09:80
  #ip: sip 10.17.170.26, dip 192.168.192.168, proto 17, dscp 0, dont fragment, last fragment, fragment offset 0
    #udp: sport 49529 dport 53 len 50
      #dns: message-type: standard query, txn id: 4
             domain name: 37.25.20.10.in-addr.arpa, type: PTR(12), class: IN(1)
[asap_firewall_forward(5218):vlan decision] len 84, vlan 1, egress CP, ingress br0:
[asap_firewall_forward(5894):bridge section] len 84, vlan 1, egress CP, ingress br0:
[asap_firewall_forward(6023):session section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6217):fastpath returned 1 opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6249):slowpath section: opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6485):back to fastpath, opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6023):session section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6217):fastpath returned 1 opcode 7] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6249):slowpath section: opcode 7] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6485):back to fastpath, opcode 3] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6789):route section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6839):cp route section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(7122):forward section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(7439):forwarding packet to bond0] len 84, vlan 1, egress bond0, ingress bond0:


Received packet from br0 (timestamp (1970-0-1 08:57:34:211376) )
[asap_firewall_forward(5048):firewall entry] len 84, vlan 0, egress CP, ingress br0:
  #mac: etype 0800 smac 6c:f3:7f:c5:2e:dc dmac 00:0b:86:86:09:80
  #ip: sip 10.17.170.26, dip 192.168.192.168, proto 17, dscp 0, dont fragment, last fragment, fragment offset 0
    #udp: sport 49531 dport 53 len 50
      #dns: message-type: standard query, txn id: 41
             domain name: device.arubanetworks.com, type: A(1), class: IN(1)
[asap_firewall_forward(5218):vlan decision] len 84, vlan 1, egress CP, ingress br0:
[asap_firewall_forward(5894):bridge section] len 84, vlan 1, egress CP, ingress br0:
[asap_firewall_forward(6023):session section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6217):fastpath returned 1 opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6249):slowpath section: opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6485):back to fastpath, opcode 4] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6023):session section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6217):fastpath returned 1 opcode 7] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6249):slowpath section: opcode 7] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6485):back to fastpath, opcode 3] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6789):route section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(6839):cp route section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(7122):forward section] len 84, vlan 1, egress bond0, ingress br0:
[asap_firewall_forward(7439):forwarding packet to bond0] len 84, vlan 1, egress bond0, ingress bond0:

Rajaguru-IAP-109# 

An uplink packet capture on the IAP can also the taken. However the DNS debug would be easier to verify this.  

To prevent the delay during the SSH login, the configured DNS server IP should be reachable by the IAP or there should not be any DNS server IAP configured on the IAP.