Security

Reply
Super Contributor II
Posts: 387
Registered: ‎09-05-2012

CPPM Slow replication, and communication issues

[ Edited ]

Hi,

 

I was just wondering, in a situation where you have both the management interface configured and the data port interface configured, which is interface is used to transfer the different kinds of data produced by the CPPM?

 

I know that the data interface is used to handle all client communnication (Onboarding, radius, etc).

For cluster replication data, which interface is used to send data between the Publisher and the Subscriber?

 

Thank you,

 

Cheers

MVP
Posts: 4,271
Registered: ‎07-20-2011

Re: CPPM Interface Usage

2015-12-02 13_39_18-Microsoft Edge.png

Thank you

Victor Fabian
Lead Mobility Engineer @ Integration Partners
AMFX | ACMX | ACDX | ACCX | CWAP | CWDP | CWNA
Guru Elite
Posts: 8,458
Registered: ‎09-08-2010

Re: CPPM Interface Usage

Take a look here:

CPPM Service Routing TechNote - V3

https://support.arubanetworks.com/Documentation/tabid/77/DMXModule/512/Command/Core_Download/Default.aspx?EntryId=14011

Sent from Nine

Tim Cappalli | Aruba Security TME
@timcappalli | timcappalli.me | ACMX #367 / ACCX #480
Moderator
Posts: 485
Registered: ‎11-09-2012

Re: CPPM Interface Usage

Take a look at this technote I wrote that cover in most what you should require.

 

CPPM Service Routing TechNote - V3

 

Its found where all my other TechNotes are located........

 

https://support.arubanetworks.com/Documentation/tabid/77/DMXModule/512/Default.aspx?EntryId=7961

 

 

 


Best Regards
-d

Snr Tech Marketing Engineer - ClearPass

-- Found something helpful, important, or cool? Click the Kudos Star in a post.
-- Problem Solved? Click "Accept as Solution" in a post.
Super Contributor II
Posts: 387
Registered: ‎09-05-2012

Re: CPPM Interface Usage

I found a KB that talks about how data is transfers here in this post.

It references the interfaces used when sending data, but what about the receiving interface on the destination CPPM?

 

We are trying to understand this to help answer some new issues that have been arising with our global cluster implementation. 

MVP
Posts: 4,271
Registered: ‎07-20-2011

Re: CPPM Interface Usage

What do you mean by this:
"It references the interfaces used when sending data, but what about the receiving interface on the destination CPPM?"

Are you load balancing your traffic ?
- From the controller
- Do you have a load balancer in front of your CPPM servers

Thank you

Victor Fabian
Lead Mobility Engineer @ Integration Partners
AMFX | ACMX | ACDX | ACCX | CWAP | CWDP | CWNA
Super Contributor II
Posts: 387
Registered: ‎09-05-2012

Re: CPPM Interface Usage

Thanks guys so much for the response! I was in the middle of writing a response for to long and didn't notice.

 

The KB makes things clearer!

 

With this knowledge, it makes the issues we are experiencing even stranger. 

We have two routes between one of our global locations. When we route traffic using route A, the publisher and subscriber see each other and replication occurs (albeit, slowly). When we route traffic using route B, the CPPM in our remote location loses contact with the Publisher CPPM. The two servers can ping each other, but to the subscriber the publisher is completely offline. The publisher though does not report the subscriber is down.

 

We tested to ensure routing and firewall rules are okay. Everything looks okay, but still the issue exists.

 

We are not using any load balancing anywhere. We have no VIP.

 

Sorry for not explaining myself properly.

 

In the case where, a publisher sends replicated data to the subscriber, that data would leave the management interface on the publisher and enter the subscriber on the subscribers management interface? Is this correct?

MVP
Posts: 1,413
Registered: ‎11-30-2011

Re: CPPM Interface Usage


bourne wrote:

In the case where, a publisher sends replicated data to the subscriber, that data would leave the management interface on the publisher and enter the subscriber on the subscribers management interface? Is this correct?


not sure if you are still looking into this, but i don't believe we can predict how it goes without knowing your full setup.

 

from the technote: In    reference    to    clustering    traffic,    the    management    IP    address    of    the    publisher    needs    to    be    accessible    to    all    subscribers.    The    subscribers    may    reach    the    publisher’s    management    IP    either    through    the    subscriber’s    management    interface    or    data    interface    based    on    network    routing    set    up.

 

so it depends on how you route the traffic. in the general that is how it works with CPPM, it does what you tell it to do.

Super Contributor II
Posts: 387
Registered: ‎09-05-2012

Re: CPPM Interface Usage

I am still working on this yes.

I am still unable to explain the behavior.

 

We have done a bunch of different tests on the routes and we have no idea why connectivity is lost when switching routes. 

 

There must be something configured incorrectly somewhere, but given this issue is only occuring between the publisher and subscriber it is hard to pin down what could possibly be the cause.

 

All this leads back to the issues we are experiencing with replication. I have asked our reseller to get Aruba support involved to help validate our setup. I want to make sure that I haven't done something stupid that is causing these issues.

Super Contributor II
Posts: 387
Registered: ‎09-05-2012

Re: CPPM Interface Usage - One way communication issue

[ Edited ]

I have an update on this issue.

 

There are two problems that eventually came out of all of this.

  1. Replication between our two sites was very slow
  2. When we switch between the available routes, the subscriber loses all access to the publisher

Issue 1 resolution

  • It is still early, but I believe this issue has been resolved by increasing the "Replication Batch Interval"
  • This was recommended to us by an Aruba tech
  • Basically what appeared to be happening was that because the latency between the two sites is high, the subscriber could keep up only during none peak times. During high peak times it would slowly fall behind and would almost never be able to catch up. By increasing the replication interval, replication occurs less frequenctly with more changes being added to the batch. The trade off is though there is more time available for the batch to be sent and processed from the publisher to the subscriber (someone please correct me if I am wrong on this!)

Issue 2 resolution

  • Full disclosure, we are still waiting on confirmation from Aruba on this
  • After a bunch of testing from the commandline (performed by an Aruba tech) we determined that it appeared as though no traffic from the subscriber going to the publisher was actually making it to the publisher. I say "appeared" because it was difficult for us to validate this 100%. The wireshark captures we took appeared so show that in fact some traffic was making, but yet the subscriber was still reporting the publisher as down.
  • My initial thought that it was one of our core switch that for some reason wasn't forwarding the traffic. The thing that was perplexing though was that the two devices could ping each other.
  • We eventually decided to try and reset the routing table on both the publisher and subscriber using the following command: network ip reset
  • After issue this command on both the  publisher and subscriber, communication was restored, much to the amazement of everyone
  • There was one key route that was removed from the routing table, the the Aruba tech wasn't sure of what it was, and this is what we are waiting on confirmation of. The routing table below
  • 0: from all lookup local
    220: from all lookup 220
    10020: from all to x.x.x.x/24 lookup mgmt
    10040: from x.x.x.x lookup mgmt
    10060: from x.x.x.x lookup data
    32766: from all lookup main
    32767: from all lookup default
  • The bolded line was the only line removed after the network ip reset, and none of use knew what the line was doing or where it came.

Sorry for the long winded reply, I just wanted to share our experience.

 

Once I get additional details from Aruba, I will update this thread.

Search Airheads
Showing results for 
Search instead for 
Did you mean: