Security

last person joined: 9 hours ago 

Enterprise security using ClearPass Policy Management, ClearPass Security Exchange, IntroSpect, VIA, 360 Security Exchange, Extensions and Policy Enforcement Firewall (PEF).
Expand all | Collapse all

Clerpasss 6.8 Clustering Problem on ESXI 6.7

Jump to Best Answer
  • 1.  Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 01, 2019 07:22 AM

    Hi There 

    I've got 2x instances of Clearpass 6.8, both servers are hosted by ESXI 6.7 

    Both Servers are in the same subnet and can be pinged on both ends from Clearpass CLI's 

    When im trying to add 2nd server as a subscriber ive got following Warning massage saying echo GET failed.

    Has anyone expierienced similliar issue before ?

     

    Regards 

    Jack 



  • 2.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 01, 2019 07:26 AM
    Is there a firewall between the clearpass nodes?


  • 3.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 01, 2019 10:39 AM

    CPPM cluster (subscriber-publisher)

    • UDP Port 123 NTP (Subscriber to publisher)
    • TCP Port 443 HTTPS (Bi-directional)
    • TCP Port 5432 PostgreSQL for DB replication (Subscriber to publisher)
    • TCP Port 80 HTTP (Between Nodes)
    • TCP Port 4231 NetWatch (Post Authentication module and the node where Insight is enabled

    check if all this ports are opened



  • 4.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 01, 2019 11:22 AM

    Hi There 

    Thanks for coming back to me on this. 

    Both Clearpass Servers are hosted by the same ESXI host 

    Also both servers are connected to the same Vswitch which is part of the same Vlan connected to Aruba Switch, there is no firewall in between the servers.

    Are there any other ways to troubleshoot this issue ?

    Many Thanks 

    Jack 

     



  • 5.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 01, 2019 02:51 PM
    Are the HTTPS certificates valid and is the time correct on both machines?


  • 6.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jun 16, 2019 05:01 AM

    HI There 

    HTTPS Certificates are on default (Out of the box) and Yes time is correct on both platforms 

     

    Best Regards 

    Jack 



  • 7.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jun 16, 2019 09:21 AM

    Make sure you using exact same software release on both nodes, is NTP configured and use the same cluserpassword on both servers.

     

    If nothing work, pleas create a tac case.



  • 8.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jun 16, 2019 11:53 AM

    Yes, i think ive tried everythig and yes both of them are in the same version 6.8.x 

    Thanks 



  • 9.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jul 03, 2019 10:35 AM

    Hi Jack,

     

    Just wondering if you had any luck from TAC on this? I have the exact same issue on two of my boxes (on 6.8.1).

     

    Thanks


    Alexander.

     

     



  • 10.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jul 11, 2019 04:33 PM

    I have tha same problem on ESXi 6.5. My clearpass VMs are on the same subnet, have you resolved the issue?



  • 11.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jul 11, 2019 04:36 PM

    Did you check your DB cert? You may need to redo the DB cert with a SAN entry with the IP address. 

    SAN .    IP:ipaddress



  • 12.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jul 11, 2019 04:43 PM

    I have fresh installations, no installed certificates, since when do I need for  clustering change default certificates SAN configuration?



  • 13.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jul 11, 2019 04:49 PM
    As of 6.8 the DB cert is now in the cert services drop down. The cluster needs to also trust the DB cert to continue. I dont remember the whole details off the top of my head but the self signed cert has the hostname and the cluster join is trying by IP. If you put the IP in the SAN field it will join. I believe it was fixed in 6.8.1 but dont quote me. I just know a quick fix is to just use the CA in onboard and sign the DB CSR from both servers with a SAN entry with the IP address


  • 14.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jul 11, 2019 04:50 PM

    One more bug, when upgrade clearpass from 6.7.10 to 6.8. Then change active system image back to 6.7.10, clearpass lose  VM activation token.

     

    Action Status: This Activation Request Token is already in use by another instance
    Product Name: ClearPass Platform
    License Type: Permanent
    User Count: 100

     

    really?

     

     

    I hope it will be recovered after  some time



  • 15.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Aug 13, 2019 10:36 AM

    Hello all,

     

    we had exactly the same topology as written in the first post.

    We were able to join the cluster usting the console command:

    "cluster make-subscriber -V -i YOU.R.I.P"

     

    Then the cluster join worked smooth!

     

    Best regards

     



  • 16.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Aug 23, 2019 04:47 AM

    I tried this with command as you but it shown.

    "

    INFO - Check publisher connection passed
    ERROR - Cannot connect to publisher database. Common errors include incorrect password, TCP port 5432 blocked, or invalid certificate.

    "

    No blocked, ping reach to each other.

    Please advise.



  • 17.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 04, 2019 09:59 AM

    Hi

     

    I can confirm

    I had the same problem in GUI interface in CPPM 6.8.2

    but with CLI command:

     

    cluster make-subscriber -V -i <ip-address>

     

    building cluster was ok

    Don't know why from GUI there is a problem 

     

    regards

     

    Karol



  • 18.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 04, 2019 12:09 PM

    Hi Karol,

     

    I tried CLI command but it did not work.



  • 19.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 11, 2019 07:08 PM

    This worked for me on ESXi 6.5 (below)


    @kkarkowski wrote:

    Hi

     

    I can confirm

    I had the same problem in GUI interface in CPPM 6.8.2

    but with CLI command:

     

    cluster make-subscriber -V -i <ip-address>

     

    building cluster was ok

    Don't know why from GUI there is a problem 

     

    regards

     

    Karol


     



  • 20.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 14, 2019 09:25 PM

    Export the Publisher's HTTPS Cert and Database Cert into the Subscriber's Trust List. Since the export Cert is in .p12 format, please convert it to a .der format. Once that is done the clustering for 6.8.x shouldn be working.



  • 21.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 16, 2019 04:09 AM

    Hi everyone,

     

    I found the issue is on ClearPass version 6.8.0.109592 that it cannot make subscriber.

    I upgrade patch to 6.8.2.109931 file size is 941 MB.

    After rebooted, I make subscriber again and it works only in CLI. 

    For GUI, still face the same issue. Y_Y



  • 22.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7
    Best Answer

    Posted Sep 16, 2019 08:06 AM

    Hi All 

    Many thanks for all your comments 

    I can confirm that after upgrading to the latest version of Clearpass 8.2 clustering with ESXI 6.7 now works with a web gui and in CLI 

    Problem Solved !

     

    Regards 

    Jack 



  • 23.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 24, 2019 01:58 PM

    Hi - on 6.8.2 here, and had to use CLI method to get it to work. Odd...

     



  • 24.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 25, 2019 12:48 AM

    It is not odd. Check release notes, HTTPS certificate validation was made mandatory for securing cluster communications and is recommended. Using CLI ignores HTTPS certificate validation.



  • 25.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 25, 2019 03:01 AM

    I think it odd. Because I drop subscriber after see the issue of failover.

    After it's standalone I tried to make-subscriber again via CLI and GUI. I stuck and got failed to do it.

    Please advise.



  • 26.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 25, 2019 03:38 AM

    if you tried from the CLI using -V option it will not fail. If it does, please work with TAC.



  • 27.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 25, 2019 03:43 AM

    First time it works but the second time it did not work.



  • 28.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Sep 25, 2019 07:17 AM

    Not sure what you tried but I just dropped my node and added it again with

     

    [appadmin@AB-CPPM-2]# cluster make-subscriber -V -i <Pub IP> 

     

    and it worked fine.



  • 29.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Nov 29, 2019 07:42 AM

    Usage:
    make-subscriber -i <IP Address> [-l] [-b] [-V]

    -i <IP Address> -- Publisher IP Address
    -l -- Restore the local log database after this operation
    -b -- Skip generating a backup before this operation
    -V -- Do not verify publisher certificate

     

    For lab/non-prod enviroment the CLI Option is fine!



  • 30.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jan 23, 2020 04:40 PM

    I have a customer experiencing this same thing via CLI and GUI on 6.8.4. 

    Output from CLI: 

    [appadmin@clearpass2]# cluster make-subscriber -V -i 192.168.16.72 -b
    
    <.....>
    
    Setting up local machine as a subscriber to 192.168.16.72
    WARNING - 192.168.16.72: echo GET failed. Will retry...
    WARNING - 192.168.16.72: echo GET failed. Will retry...
    ERROR - Publisher connection failed
    ERROR - Connection to publisher failed. Please check that:
    ERROR -  1) Publisher IP address and cluster password is valid and synchronized
    ERROR -  2) Publisher is up and accessible from this machine
    ERROR -  3) License is active
    ERROR - Setting up subscriber failed

    I plan on calling TAC tomorrow. 

    *** EDIT ***: After further troubleshooting I found that the Cluster Password did not match the appadmin password somehow?? I went through the "Change Cluster Password" workflow, and CLI cluter join worked fine (with -V switch). Very strange.



  • 31.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jan 24, 2020 03:38 AM
      |   view attached

    In 6.8.0 the process of clustering ClearPass Policy Manager nodes requires 2 certificate validations.
    a. HTTPS certificate validation
    b. Database certificate validation

     

    I have attached cluster doucment, add publisher root CA in subcriber trust list.

    Attachment(s)

    pdf
    Clustering in 6.8.pdf   985K 1 version


  • 32.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Jan 24, 2020 05:54 AM

    Hi Zmerick1,

     

    After further troubleshooting I found that the Cluster Password did not match the appadmin password somehow?? I went through the "Change Cluster Password" workflow, and CLI cluster join worked fine (with -V switch). Very strange.

     

    Clarification:

     

    when we add the subscriber to the publisher ( I  Mean when we form the cluster ) we insert entry in the publisher database for the subscriber. The only way to get it into the publisher database is though the appadmin (cluster password).

     

    sometimes while updating the cluster password from GUI will not get updated in the database (could be issue with http frontend process running from a long time) and upon getting the request from the subscriber to add the new entry in the database with password it will not allow.

     

    This could also happen if the cluster password have some special characters and while updating them again has accepted by the postgresql database. 

     

    As mentioned by @Pavan above , Both HTTPS and Database certificate trust is also required. which you have ignored with -V option.

     

    [appadmin@vikram-cppm]# cluster make-subscriber <Enter>

     

    Usage:
    make-subscriber -i <IP Address> [-l] [-b] [-V]

    -i <IP Address> -- Publisher IP Address
    -l -- Restore the local log database after this operation
    -b -- Skip generating a backup before this operation
    -V -- Do not verify publisher certificate


    [appadmin@vikram-cppm]#

     

     



  • 33.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 06, 2020 12:45 AM

    I am experiencing this issue in 6.9.0 - not fixed!

    Same error in GUI and CLI

    I am using default certificates and cli with -V option.

     

    [appadmin@cppm02]# cluster make-subscriber -V -i a.b.c.d
    
    ********************************************************
    *                                                      *
    * WARNING: Executing this command will make the current*
    * machine a subscriber to the publisher host specified.*
    * Current configuration and application licenses       *
    * installed (if any) on this node will be lost when the*
    * operation is complete.                               *
    *                                                      *
    * Configuration changes will be blocked on the         *
    * publisher during initial cluster sync as part of     *
    * this operation.                                     *
    *                                                      *
    * Do not close the shell or interrupt this command     *
    * execution.                                           *
    *                                                      *
    ********************************************************
    
    Continue? [y|n]: y
    
    Enter Publisher Password:
    Setting up local machine as a subscriber to  a.b.c.d
    WARNING - a.b.c.d: echo GET failed. Will retry...
    WARNING - a.b.c.d: echo GET failed. Will retry...
    ERROR - Publisher connection failed
    ERROR - Connection to publisher failed. Please check that:
    ERROR -  1) Publisher IP address and cluster password is valid and synchronized
    ERROR -  2) Publisher is up and accessible from this machine
    ERROR -  3) License is active
    ERROR - Setting up subscriber failed
    
    [appadmin@cppm02]#

     



  • 34.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 06, 2020 04:23 AM
      |   view attached

    I was able to do this by importing BOTH the HTTPS and Database certs of the Publisher to the subscribers' Trust Lists and joining via the GUI.  See attached guide on how to do it.

     

    It does appear the -V paramater in the CLI command doesn't work as intended to bypass certificate verification.

    Attachment(s)

    pdf
    Clustering in 6.8.pdf   985K 1 version


  • 35.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 06, 2020 05:41 AM

    In the long run, we should remove the -v option. This is to cultivate the correct usage of the certificates for clustering. I do not use it but I follow the clustering Technote and was able to do it in GUI for 6.8 a couple of times. I doubt there is a difference in 6.9. 



  • 36.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 06, 2020 09:48 AM

    I wish Aruba would put in "Break Changes" in the Release Notes. As a customer and a partner I can tell you that most people don't read the release notes for behavior changes. Customers want the latest features,or they are directed to bump the code by TAC to solve issue XYZ. Which, in this case, could break things. I know this is not an Aruba issue per-se, but I'm just stating what I see in the wild. It would be helpful to have Break Changes highlighted.

     

    With all that being said, I'm on the fence about removing the -v option. As it will provide a safe-guard for customers that are having issues with the whole process. The security implications of the subscriber having a bad trust with the publisher, or a server impersonating a subscriber is next to nil. On the other hand . . . . zero-trust. Maybe it is slowly deprecated, because by the views of this post this "issue" has hit a lot of people.



  • 37.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted May 07, 2020 08:35 AM

    Closing this thread because it is resolved.  If you have any feature requests, please post in https://innovate.arubanetworks.com

     

    Please also take a look at the Clustering Design Guidelines Tech Note here for more clustering information:  https://support.arubanetworks.com/Documentation/tabid/77/DMXModule/512/Default.aspx?EntryId=33093

     

     



  • 38.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Dec 23, 2019 04:06 PM

    @Same Guy wrote:

    Export the Publisher's HTTPS Cert and Database Cert into the Subscriber's Trust List. Since the export Cert is in .p12 format, please convert it to a .der format. Once that is done the clustering for 6.8.x shouldn be working.

     

    I converted HTTPS .p12 into .pem and imported to subscribers trust list. clustering worked well. I did this on 6.8.0


     



  • 39.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Feb 07, 2020 02:51 PM

    I just came across this issue. This worked for me (!!)

     

    So weird and I'm on 6.8.4.

     

    Definitely a bug!



  • 40.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Feb 11, 2020 01:11 PM

    Ran into this issue on a new cluster deployment on two hardware servers running 6.8.4.120034.

    I had Initially created the cluster using the cli command with the -V argument.

    A day later the subscriber went out of sync. I proceeded to drop the subscriber and re-add to the cluster but it failed this time even with the cli.

    I've signed the publisher's certificate and imported into the subscriber's trust list but it still didn't join the cluster.

    I've now been working with TAC for 3 days now and none of the action plans to rejoin the cluster has worked. TAC has proceeded to involve the engineering team.



  • 41.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Feb 13, 2020 06:04 AM

    Can you please share TAC ticket number, if needed we will elevate this case to next level.

     



  • 42.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Feb 13, 2020 06:48 AM

    Pavan,

     

    HPE Support Case 5344999338

     

    Thanks



  • 43.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Feb 13, 2020 07:02 AM

    We have received work around from dev team and I believe you are on session with TAC engineer now to apply workaround.



  • 44.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Feb 13, 2020 07:25 AM

    Workaround will fix only restore issue, still cluster part is pending. Pushing for immediate fix from dev.



  • 45.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Feb 13, 2020 07:57 AM

    The workaround for restore worked. Now waiting for the cluster fix.

     

    Thanks for the follow up.



  • 46.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Mar 30, 2020 05:42 AM

    Has this issue been resolved yet? I am facing the same problem.



  • 47.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Mar 30, 2020 06:11 AM

    Cluster Join issue also been fixed, need to change file permission in subscriber to sync with publisher. Please open TAC ticket.



  • 48.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Mar 30, 2020 06:12 AM

    Great thank you. Will open a ticket.



  • 49.  RE: Clerpasss 6.8 Clustering Problem on ESXI 6.7

    Posted Mar 30, 2020 07:37 AM

    Oh yes. You would need TAC to help with the file permission changes.