Question : Why do we see Airwave user fails to SSH sometimes while Auditing the Controller?
Environment Information : Any Aruba Controller. Any Aruba OS, Any Airwave version
Symptoms : Admin user fails to SSH sometimes while auditing their controller.
Jan 20 23:45:49 :124004: <DBUG> |authmgr| unknown user=10.21.0.1, method=Management
Jan 20 23:45:49 :124547: <DBUG> |authmgr| aal_authenticate server_group:default.
Jan 20 23:45:49 :124004: <DBUG> |authmgr| Select server for method=Management, user=admin, essid=<>, server-group=Management, last_srv <>
Jan 20 23:45:49 :124004: <DBUG> |authmgr| server=Internal, ena=1, ins=1 (1)
Jan 20 23:45:49 :124038: <INFO> |authmgr| Selected server Internal for method=Management; user=admin, essid=<>, domain=<>, server-group=Management
Jan 20 23:45:49 :133004: <INFO> |localdb| Received Authentication Request for User admin
Jan 20 23:45:49 :133019: <ERRS> |localdb| User admin was not found in the database
Jan 20 23:45:49 :133006: <ERRS> |localdb| User admin Failed Authentication
Jan 20 23:45:49 :124230: <DBUG> |authmgr| Rx message 21/23, length 351 from 127.0.0.1:8344
Jan 20 23:45:49 :124004: <DBUG> |authmgr| Local DB auth failed for user admin, error (User not found in UserDB)
Jan 20 23:45:49 :124064: <NOTI> |authmgr| Administrative User result=Authentication failed(1), method=Management, username=admin IP=10.21.0.1 auth server=Internal
Jan 20 23:45:49 :124003: <INFO> |authmgr| Authentication result=Authentication failed(1), method=Management, server=Internal, user=10.21.0.1
Jan 20 23:45:49 :124004: <DBUG> |authmgr| Auth server 'Internal' response=1
Jan 20 23:45:49 :124004: <DBUG> |authmgr| Select server for method=Management, user=admin, essid=<>, server-group=Management, last_srv Internal
Jan 20 23:45:49 :124004: <DBUG> |authmgr| server=Clearpass, ena=1, ins=1 (1)
Jan 20 23:45:49 :124038: <INFO> |authmgr| Selected server Clearpass for method=Management; user=admin, essid=<>, domain=<>, server-group=Management
Jan 20 23:45:49 :124545: <DBUG> |authmgr| Fail-thru to Clearpass.
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_api.c:383] Radius authenticate user (admin) PAP using server Clearpass
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_api.c:1173] :L2 User lookup failed, skipping Aruba-Port-ID
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_request.c:52] Add Request: id=3, srv=10.10.105.30, fd=77
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1254] Sending radius request to Clearpass:10.10.105.30:1812 id:3,len:156
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] NAS-IP-Address: 10.21.0.64
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] NAS-Port-Id: 0
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] NAS-Port-Type: 5
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] User-Name: admin
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1268] Password: *****
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] Service-Type: Administrative-User
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] Calling-Station-Id: 10.21.0.1
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] Called-Station-Id: 000B8661DF3C
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] Framed-IP-Address: 10.21.0.1
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] Aruba-Essid-Name:
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] Aruba-Location-Id: N/A
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] Aruba-AP-Group: N/A
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] Aruba-Device-Type:
Jan 20 23:45:49 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:1264] Message-Auth: \263\257b\314\373e\337\304\030\033\307\333&\225\267\276
Jan 20 23:45:52 :137031: <DBUG> |mdns| [seq_num_timeout_handler:121] seq_num_timeout_handler: Freed 0 entries
Jan 20 23:45:54 :121004: <WARN> |authmgr| |aaa| RADIUS server Clearpass--10.10.105.30-1812 timeout for client=00:00:00:00:00:00 auth method Management
Jan 20 23:45:54 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:775] Sending radius request to Clearpass-10.10.105.30-1812 (retry 1)
Jan 20 23:45:55 :121031: <DBUG> |authmgr| |aaa| [rc_sequence.c:111] seq_num_timeout_handler: Freed 0 entries
Jan 20 23:45:59 :121004: <WARN> |authmgr| |aaa| RADIUS server Clearpass--10.10.105.30-1812 timeout for client=00:00:00:00:00:00 auth method Management
Jan 20 23:45:59 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:775] Sending radius request to Clearpass-10.10.105.30-1812 (retry 2)
Jan 20 23:46:02 :137031: <DBUG> |mdns| [seq_num_timeout_handler:121] seq_num_timeout_handler: Freed 0 entries
Jan 20 23:46:04 :121004: <WARN> |authmgr| |aaa| RADIUS server Clearpass--10.10.105.30-1812 timeout for client=00:00:00:00:00:00 auth method Management
Jan 20 23:46:04 :121031: <DBUG> |authmgr| |aaa| [rc_server.c:775] Sending radius request to Clearpass-10.10.105.30-1812 (retry 3)
Jan 20 23:46:05 :121031: <DBUG> |authmgr| |aaa| [rc_sequence.c:111] seq_num_timeout_handler: Freed 0 entries
Jan 20 23:46:09 :121004: <WARN> |authmgr| |aaa| RADIUS server Clearpass--10.10.105.30-1812 timeout for client=00:00:00:00:00:00 auth method Management
Jan 20 23:46:09 :121031: <DBUG> |authmgr| |aaa| [rc_request.c:37] Del Request: id=3, srv=10.10.105.30, fd=77
Jan 20 23:46:09 :121031: <DBUG> |authmgr| |aaa| [rc_api.c:1029] AAA server timeout
Jan 20 23:46:09 :124064: <NOTI> |authmgr| Administrative User result=AAA server timeout(2), method=Management, username=admin IP=10.21.0.1 auth server=Clearpass
Jan 20 23:46:09 :124003: <INFO> |authmgr| Authentication result=AAA server timeout(2), method=Management, server=Clearpass, user=10.21.0.1
Jan 20 23:46:09 :124004: <DBUG> |authmgr| Auth server 'Clearpass' response=2
Jan 20 23:46:09 :124014: <NOTI> |authmgr| Taking Server Clearpass out of service for 10 mins
Jan 20 23:46:09 :124004: <DBUG> |authmgr| Select server for method=Management, user=admin, essid=<>, server-group=Management, last_srv Clearpass
Jan 20 23:46:09 :124038: <INFO> |authmgr| Selected server <> for method=Management; user=admin, essid=<>, domain=<>, server-group=Management
Jan 20 23:46:09 :124544: <DBUG> |authmgr| Timed Out to N/A.
Jan 20 23:46:09 :124541: <DBUG> |authmgr| Bring all servers in server group Management back in service.
Jan 20 23:46:09 :124015: <NOTI> |authmgr| Bringing Server Clearpass back in service.
Jan 20 23:46:09 :125027: <DBUG> |aaa| mgmt-auth: admin, failure, , 0
Jan 20 23:46:09 :125024: <NOTI> |aaa| Authentication Succeeded for User admin, Logged in from 10.21.0.1 port 52633, Connecting to 10.21.0.64 port 22 connection type SSH
Cause : Management authentication includes a server-group with Internal and Radius server in the server-group for mgmt-login and radius authentication for mgmt is enabled.
Resolution : Verify the configuration of management authentication and the server group configured. If we see a lot of failures for "admin" user, it is likely because "Radius authentication" is enabled and hence Controllers checks for the admin user in all the servers in the group before kicking in local-auth for mgmt-user.
Answer :
The following is a possible root cause for such an issue:
1.Management authentication includes a server-group with Internal and Radius server in the server-group for mgmt-login and radius authentication for mgmt is enabled. Hence the login request is first sent to “Internal dB”.
2.Since there are no entries in the local-userdb to match the login username/password for “admin” and fail-through is enabled, requests are sent to radius server which is timing out.
3.Since we also have "no mgmt-user localauth-disable", the “admin” login request is then matched with the mgmt-user entries and hence login is successful.
4.If we have radius authentication enabled i.e.:
aaa authentication mgmt
server-group "Management"
enable<<<<<<<<<<<<<<<<<<<<<<<
Then the sequence of authentication check for “admin” user login request is as below:
a)“Internal” server that is marked in position 1 of server-group “Management”.
b)“Clearpass” server marked in position 2 of the server-group “Management”.
c)If a and b fail, then check for the “mgmt-user admin root 78edd4cd01cb96c861f6671cd6e8b524e7773735a83724cc73”
Resolution:
1)If we want login requests to bypass mgmt-server-group and go to mgmt-user ‘admin’ only then please do "no enable" on " aaa authentication mgmt".
This is the only way to give mgmt-user higher preference over the server-group for mgmt-login-request.
(OR)
2)Add the “admin” user to local user database on the Master Controller. [need to add to local Controllers too if the knob “Use Local Switch’s Database for Authentication” on Internal Database is enabled]
Example:
(Aruba) #local-userdb add username "admin" password ****** role "root”
(Aruba) #write mem
(Aruba) #show local-userdb
User Summary
------------
Name Password Role E-Mail Enabled Expiry Status Sponsor-Name Remote-IP Grantor-Name
---- -------- ---- ------ ------- ------ ------ ------------ --------- ------------
guest451229 ******** guest Yes 3/12/2014 15:00 Active 0.0.0.0 helpdesk
admin ******** root Yes Active 0.0.0.0 admin
User Entries: 2
(Aruba) #show log security 30
Jan 22 04:27:24 :124004: <DBUG> |authmgr| unknown user=10.20.25.58, method=Management
Jan 22 04:27:24 :124547: <DBUG> |authmgr| aal_authenticate server_group:default.
Jan 22 04:27:24 :124004: <DBUG> |authmgr| Select server for method=Management, user=admin, essid=<>, server-group=test, last_srv <>
Jan 22 04:27:24 :124004: <DBUG> |authmgr| server=Internal, ena=1, ins=1 (1)
Jan 22 04:27:24 :124038: <INFO> |authmgr| Selected server Internal for method=Management; user=admin, essid=<>, domain=<>, server-group=test
Jan 22 04:27:24 :133004: <INFO> |localdb| Received Authentication Request for User admin
Jan 22 04:27:24 :133005: <INFO> |localdb| User admin root Successfully Authenticated
Jan 22 04:27:24 :124230: <DBUG> |authmgr| Rx message 21/22, length 2995 from 127.0.0.1:8344
Jan 22 04:27:24 :124066: <INFO> |authmgr| Administrative User result=Authentication Successful(0), method=Management, username=admin IP=10.20.25.58 auth server=Internal
Jan 22 04:27:24 :124003: <INFO> |authmgr| Authentication result=Authentication Successful(0), method=Management, server=Internal, user=10.20.25.58
Jan 22 04:27:24 :124004: <DBUG> |authmgr| Auth server 'Internal' response=0
Jan 22 04:27:24 :124025: <NOTI> |authmgr| Administrative user 'admin' authenticated successfully (role=root, privileged=0)
Jan 22 04:27:24 :125027: <DBUG> |aaa| mgmt-auth: admin, success, root, 0
Jan 22 04:27:24 :125024: <NOTI> |aaa| Authentication Succeeded for User admin, Logged in from 10.20.25.58 port 54298, Connecting to 10.20.25.102 port 22 connection type SSH
This explains why we see failures on Clearpass when Airwave is logging in with “admin” credentials.