Security

Hi There 

I've got 2x instances of Clearpass 6.8, both servers are hosted by ESXI 6.7 

Both Servers are in the same subnet and can be pinged on both ends from Clearpass CLI's 

When im trying to add 2nd server as a subscriber ive got following Warning massage saying echo GET failed.

Has anyone expierienced similliar issue before ?

 

Regards 

Jack 

Is there a firewall between the clearpass nodes?

CPPM cluster (subscriber-publisher)

• UDP Port 123 NTP (Subscriber to publisher)
• TCP Port 443 HTTPS (Bi-directional)
• TCP Port 5432 PostgreSQL for DB replication (Subscriber to publisher)
• TCP Port 80 HTTP (Between Nodes)
• TCP Port 4231 NetWatch (Post Authentication module and the node where Insight is enabled

check if all this ports are opened

Hi There 

Thanks for coming back to me on this. 

Both Clearpass Servers are hosted by the same ESXI host 

Also both servers are connected to the same Vswitch which is part of the same Vlan connected to Aruba Switch, there is no firewall in between the servers.

Are there any other ways to troubleshoot this issue ?

Many Thanks 

Jack 

 

Are the HTTPS certificates valid and is the time correct on both machines?

HI There 

HTTPS Certificates are on default (Out of the box) and Yes time is correct on both platforms 

 

Best Regards 

Jack 

Make sure you using exact same software release on both nodes, is NTP configured and use the same cluserpassword on both servers.

 

If nothing work, pleas create a tac case.

Yes, i think ive tried everythig and yes both of them are in the same version 6.8.x 

Thanks 

Hi Jack,

 

Just wondering if you had any luck from TAC on this? I have the exact same issue on two of my boxes (on 6.8.1).

 

Thanks

Alexander.

 

 

I have tha same problem on ESXi 6.5. My clearpass VMs are on the same subnet, have you resolved the issue?

Did you check your DB cert? You may need to redo the DB cert with a SAN entry with the IP address. 

SAN .    IP:ipaddress

I have fresh installations, no installed certificates, since when do I need for  clustering change default certificates SAN configuration?

As of 6.8 the DB cert is now in the cert services drop down. The cluster needs to also trust the DB cert to continue. I dont remember the whole details off the top of my head but the self signed cert has the hostname and the cluster join is trying by IP. If you put the IP in the SAN field it will join. I believe it was fixed in 6.8.1 but dont quote me. I just know a quick fix is to just use the CA in onboard and sign the DB CSR from both servers with a SAN entry with the IP address

One more bug, when upgrade clearpass from 6.7.10 to 6.8. Then change active system image back to 6.7.10, clearpass lose  VM activation token.

 

Action Status: This Activation Request Token is already in use by another instance
Product Name: ClearPass Platform
License Type: Permanent
User Count: 100

 

really?

 

 

I hope it will be recovered after  some time

Hello all,

 

we had exactly the same topology as written in the first post.

We were able to join the cluster usting the console command:

"cluster make-subscriber -V -i YOU.R.I.P"

 

Then the cluster join worked smooth!

 

Best regards

 

I tried this with command as you but it shown.

"

INFO - Check publisher connection passed
ERROR - Cannot connect to publisher database. Common errors include incorrect password, TCP port 5432 blocked, or invalid certificate.

"

No blocked, ping reach to each other.

Please advise.

Hi

 

I can confirm

I had the same problem in GUI interface in CPPM 6.8.2

but with CLI command:

 

cluster make-subscriber -V -i <ip-address>

 

building cluster was ok

Don't know why from GUI there is a problem 

 

regards

 

Karol

Hi Karol,

 

I tried CLI command but it did not work.

This worked for me on ESXi 6.5 (below)

---

@kkarkowski wrote:

Hi

 

I can confirm

I had the same problem in GUI interface in CPPM 6.8.2

but with CLI command:

 

cluster make-subscriber -V -i <ip-address>

 

building cluster was ok

Don't know why from GUI there is a problem 

 

regards

 

Karol

---

 

Export the Publisher's HTTPS Cert and Database Cert into the Subscriber's Trust List. Since the export Cert is in .p12 format, please convert it to a .der format. Once that is done the clustering for 6.8.x shouldn be working.

Hi everyone,

 

I found the issue is on ClearPass version 6.8.0.109592 that it cannot make subscriber.

I upgrade patch to 6.8.2.109931 file size is 941 MB.

After rebooted, I make subscriber again and it works only in CLI. 

For GUI, still face the same issue. Y_Y

Hi All 

Many thanks for all your comments 

I can confirm that after upgrading to the latest version of Clearpass 8.2 clustering with ESXI 6.7 now works with a web gui and in CLI 

Problem Solved !

 

Regards 

Jack 

Hi - on 6.8.2 here, and had to use CLI method to get it to work. Odd...

 

It is not odd. Check release notes, HTTPS certificate validation was made mandatory for securing cluster communications and is recommended. Using CLI ignores HTTPS certificate validation.

I think it odd. Because I drop subscriber after see the issue of failover.

After it's standalone I tried to make-subscriber again via CLI and GUI. I stuck and got failed to do it.

Please advise.

if you tried from the CLI using -V option it will not fail. If it does, please work with TAC.

First time it works but the second time it did not work.

Not sure what you tried but I just dropped my node and added it again with

 

[appadmin@AB-CPPM-2]# cluster make-subscriber -V -i <Pub IP> 

 

and it worked fine.

Usage:
make-subscriber -i <IP Address> [-l] [-b] [-V]

-i <IP Address> -- Publisher IP Address
-l -- Restore the local log database after this operation
-b -- Skip generating a backup before this operation
-V -- Do not verify publisher certificate

 

For lab/non-prod enviroment the CLI Option is fine!

I have a customer experiencing this same thing via CLI and GUI on 6.8.4. 

Output from CLI: 

[appadmin@clearpass2]# cluster make-subscriber -V -i 192.168.16.72 -b

<.....>

Setting up local machine as a subscriber to 192.168.16.72
WARNING - 192.168.16.72: echo GET failed. Will retry...
WARNING - 192.168.16.72: echo GET failed. Will retry...
ERROR - Publisher connection failed
ERROR - Connection to publisher failed. Please check that:
ERROR -  1) Publisher IP address and cluster password is valid and synchronized
ERROR -  2) Publisher is up and accessible from this machine
ERROR -  3) License is active
ERROR - Setting up subscriber failed

I plan on calling TAC tomorrow. 

*** EDIT ***: After further troubleshooting I found that the Cluster Password did not match the appadmin password somehow?? I went through the "Change Cluster Password" workflow, and CLI cluter join worked fine (with -V switch). Very strange.

In 6.8.0 the process of clustering ClearPass Policy Manager nodes requires 2 certificate validations.
a. HTTPS certificate validation
b. Database certificate validation

 

I have attached cluster doucment, add publisher root CA in subcriber trust list.

Hi Zmerick1,

 

After further troubleshooting I found that the Cluster Password did not match the appadmin password somehow?? I went through the "Change Cluster Password" workflow, and CLI cluster join worked fine (with -V switch). Very strange.

 

Clarification:

 

when we add the subscriber to the publisher ( I  Mean when we form the cluster ) we insert entry in the publisher database for the subscriber. The only way to get it into the publisher database is though the appadmin (cluster password).

 

sometimes while updating the cluster password from GUI will not get updated in the database (could be issue with http frontend process running from a long time) and upon getting the request from the subscriber to add the new entry in the database with password it will not allow.

 

This could also happen if the cluster password have some special characters and while updating them again has accepted by the postgresql database. 

 

As mentioned by @Pavan above , Both HTTPS and Database certificate trust is also required. which you have ignored with -V option.

 

[appadmin@vikram-cppm]# cluster make-subscriber <Enter>

 

Usage:
make-subscriber -i <IP Address> [-l] [-b] [-V]

-i <IP Address> -- Publisher IP Address
-l -- Restore the local log database after this operation
-b -- Skip generating a backup before this operation
-V -- Do not verify publisher certificate

[appadmin@vikram-cppm]#

 

 

I am experiencing this issue in 6.9.0 - not fixed!

Same error in GUI and CLI

I am using default certificates and cli with -V option.

 

[appadmin@cppm02]# cluster make-subscriber -V -i a.b.c.d

********************************************************
*                                                      *
* WARNING: Executing this command will make the current*
* machine a subscriber to the publisher host specified.*
* Current configuration and application licenses       *
* installed (if any) on this node will be lost when the*
* operation is complete.                               *
*                                                      *
* Configuration changes will be blocked on the         *
* publisher during initial cluster sync as part of     *
* this operation.                                     *
*                                                      *
* Do not close the shell or interrupt this command     *
* execution.                                           *
*                                                      *
********************************************************

Continue? [y|n]: y

Enter Publisher Password:
Setting up local machine as a subscriber to  a.b.c.d
WARNING - a.b.c.d: echo GET failed. Will retry...
WARNING - a.b.c.d: echo GET failed. Will retry...
ERROR - Publisher connection failed
ERROR - Connection to publisher failed. Please check that:
ERROR -  1) Publisher IP address and cluster password is valid and synchronized
ERROR -  2) Publisher is up and accessible from this machine
ERROR -  3) License is active
ERROR - Setting up subscriber failed

[appadmin@cppm02]#

 

I was able to do this by importing BOTH the HTTPS and Database certs of the Publisher to the subscribers' Trust Lists and joining via the GUI.  See attached guide on how to do it.

 

It does appear the -V paramater in the CLI command doesn't work as intended to bypass certificate verification.

In the long run, we should remove the -v option. This is to cultivate the correct usage of the certificates for clustering. I do not use it but I follow the clustering Technote and was able to do it in GUI for 6.8 a couple of times. I doubt there is a difference in 6.9. 

I wish Aruba would put in "Break Changes" in the Release Notes. As a customer and a partner I can tell you that most people don't read the release notes for behavior changes. Customers want the latest features,or they are directed to bump the code by TAC to solve issue XYZ. Which, in this case, could break things. I know this is not an Aruba issue per-se, but I'm just stating what I see in the wild. It would be helpful to have Break Changes highlighted.

 

With all that being said, I'm on the fence about removing the -v option. As it will provide a safe-guard for customers that are having issues with the whole process. The security implications of the subscriber having a bad trust with the publisher, or a server impersonating a subscriber is next to nil. On the other hand . . . . zero-trust. Maybe it is slowly deprecated, because by the views of this post this "issue" has hit a lot of people.

Closing this thread because it is resolved.  If you have any feature requests, please post in https://innovate.arubanetworks.com

 

Please also take a look at the Clustering Design Guidelines Tech Note here for more clustering information:  https://support.arubanetworks.com/Documentation/tabid/77/DMXModule/512/Default.aspx?EntryId=33093

 

 

---

@Same Guy wrote:

Export the Publisher's HTTPS Cert and Database Cert into the Subscriber's Trust List. Since the export Cert is in .p12 format, please convert it to a .der format. Once that is done the clustering for 6.8.x shouldn be working.

 

I converted HTTPS .p12 into .pem and imported to subscribers trust list. clustering worked well. I did this on 6.8.0

---

 

I just came across this issue. This worked for me (!!)

 

So weird and I'm on 6.8.4.

 

Definitely a bug!

Ran into this issue on a new cluster deployment on two hardware servers running 6.8.4.120034.

I had Initially created the cluster using the cli command with the -V argument.

A day later the subscriber went out of sync. I proceeded to drop the subscriber and re-add to the cluster but it failed this time even with the cli.

I've signed the publisher's certificate and imported into the subscriber's trust list but it still didn't join the cluster.

I've now been working with TAC for 3 days now and none of the action plans to rejoin the cluster has worked. TAC has proceeded to involve the engineering team.

Can you please share TAC ticket number, if needed we will elevate this case to next level.

 

Pavan,

 

HPE Support Case 5344999338

 

Thanks

We have received work around from dev team and I believe you are on session with TAC engineer now to apply workaround.

Workaround will fix only restore issue, still cluster part is pending. Pushing for immediate fix from dev.

The workaround for restore worked. Now waiting for the cluster fix.

 

Thanks for the follow up.

Has this issue been resolved yet? I am facing the same problem.

Cluster Join issue also been fixed, need to change file permission in subscriber to sync with publisher. Please open TAC ticket.

Great thank you. Will open a ticket.

Oh yes. You would need TAC to help with the file permission changes.