Problem:
Network Scan fails to start when primary master for zone is subscriber node.
Diagnostics:In a scenario where multiple zone are configured in cluster as below:
HOSTNAME ZONE SERVER ROLE
192.168.1.2 ASIA Publisher
192.168.1.3 ASIA Subscriber
192.168.1.4 EUROPE Subscriber
192.168.1.5 EUROPE Subscriber
Clearpass node: 192.168.1.2 is the primary master for zone: ASIA
Clearpass node: 192.168.1.4 is the primary master for zone: EUROPE
Under Monitoring->Profiler and Network Scan->Network Scan Results, network scan results would be available for zone:ASIA where the primary master is Publisher (192.168.1.2).
Network scan results are would not be available for zone: EUROPE where primary master for zone :EUROPE is Subscriber(192.168.1.4).
Collect policy manager logs of subscriber node which is primary master for particular zone where network scan is not working (In the above example, from server 192.168.1.4) to identify the issue. Place "Async network services" service in DEBUG [(Administration->Server Manager->Log Configuration. Select server(primary master of zone having issues for network scan)].
Navigate to Administration->Server Manager->Server Configuration. Select the radio button of subscriber node (primary master of zone) and click "Collect Logs". Uncheck all options and check "Logs from all Policy Manager services". Click "Start". Once completed, click "Download File"
Extract the logs and navigate to tips-network-services folder and open the network-services.log
From the network-services.log of subscriber node: 192.168.1.4 which is primary master for zone:EUROPE, we can see that the SNMP subnet scan fails to start with below error:
2020-03-29 14:10:17,149 [DefaultQuartzScheduler_Worker-4] [R:] INFO com.avenda.tips.snmpserver.network.scan.ScanJob - Seed devices=192.168.100.49 Network discovery distributed to {192.168.1.4}
2020-03-29 14:10:17,162 [pool-15-thread-28] [R:NOTSET] ERROR com.avenda.tips.snmpserver.request.HttpTask - Unexpected failure - status=401 response=<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Unauthorized</title>
</head><body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>
~_2020-03-29 14:10:17,165 [pool-15-thread-28] [R:NOTSET] ERROR com.avenda.tips.snmpserver.request.HttpTask - network-discovery request=POST url=*http://192.168.1.4/networkservices/snmpservice/internal/NetworkDiscovery body=
{"seedDevices":["192.168.100.49"],"scanDepth":1,"configId":"42","probeArp":true,"scanRunId":"discovery-42-1579612217150"}
failed reason=401-Unauthorized_~
2020-03-29 14:10:17,165 [pool-15-thread-28] [R:NOTSET] WARN com.avenda.tips.utils.async.AsyncTask - Alert: network-discovery request failed.
2020-03-29 14:10:17,165 [pool-15-thread-28] [R:NOTSET] WARN com.avenda.tips.utils.async.AsyncTask - Stop task: Task=com.avenda.tips.snmpserver.request.HttpTask@596d0428 stopped with error=network-discovery request failed. alerts=[network-discovery request failed.]
SolutionIf 401-Unauthorized error is seen while initiating network scan in logs, then that means the nodes in cluster may not have the cluster password synced which causes network scan to fail.
In order to sync cluster password to subscriber node, you can either update the cluster(appadmin) password once again or we can try the below CLI command on subscriber node in order to syncronize the cluster password with Publisher:
#cluster sync-cluster-passwd
To ensure cluster password is updated in primary master for particular zone (In this example, 192.168.1.4 which is primary master for zone: EUROPE), download the policy manager logs and verify the dbcn-daemon.log available under dbcn-daemon folder for below entry to confirm cluster password update in subscriber node.
2020-03-29 14:37:32,299 INFO Platform.DbcnDaemon AppDbcnHandlerRegistry [NetworkServicesDbcnHandler] <= DBCN:[[9844, 'ADMIN_USER', 1, 'UPDATE', None, '2020-03-29 14:37:36.557302+02:00']