Security

Hey Airheads,

I've got a new install of CPPM 6.10.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.

Curious why you have a new install of CPPM 6.5??? That version is well past support.  You should upgrade ClearPass to 6.10.
Original Message:
Sent: Aug 04, 2022 03:05 AM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hey Airheads,

I've got a new install of CPPM 6.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.

Title mentions CPPM 6.10, so may be a typo in the message.

Do you see something on the Panorama side in the logging? Have you create packet captures of the traffic between ClearPass and Panorama? Are certificates/trust for https configured and still valid?

If I freely interpret the error message, it looks like CPPM can connect to the Panorama, SSL sessions comes up (so probably certs are okay), then ClearPass sends the request, but never hears back something from Panorama. If you have the full URL, and can find the JSON/XML (think you can get that from the postauth.log file if you run a 'Collect Logs'), you could replay that command with Postman or curl/wget if you know how to do that.

My PAN integration just works with CPPM 6.10, I only have a single firewall and no Panorama. May be good to get PAN Support involved as well, as it may be Panorama acting strangely. Are you sending high numbers of userid updates (think multiple per second)?

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
------------------------------

Original Message:
Sent: Aug 04, 2022 07:29 AM
From: Unknown User
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Curious why you have a new install of CPPM 6.5??? That version is well past support.  You should upgrade ClearPass to 6.10.
Original Message:
Sent: Aug 04, 2022 03:05 AM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hey Airheads,

I've got a new install of CPPM 6.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.

hey herman, thanks for the reply. 

it was just working fine but have also started seeing other errors in the cluster so wondering if something isn't happy internally. Looking at the palo we see successful login events from ClearPass so this could be intermittent. XML data is flowing but clearpass is reporting this error so thinking it could be a timeout type of scenario. there is a very large number of firewall serials in the request so wonder if thats related. 

Source PolicyServer
Level WARN
Category HTTP
Action Failed
Timestamp Aug 04, 2022 15:35:18 AEST
Description
Unable to communicate with HTTP server http://localhost:6179/async_netd/cmdctrl

Thinking these could be related. a restart of server cleared this for a while but it seems to have returned. 


Original Message:
Sent: Aug 04, 2022 10:17 AM
From: Herman Robers
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Title mentions CPPM 6.10, so may be a typo in the message.

Do you see something on the Panorama side in the logging? Have you create packet captures of the traffic between ClearPass and Panorama? Are certificates/trust for https configured and still valid?

If I freely interpret the error message, it looks like CPPM can connect to the Panorama, SSL sessions comes up (so probably certs are okay), then ClearPass sends the request, but never hears back something from Panorama. If you have the full URL, and can find the JSON/XML (think you can get that from the postauth.log file if you run a 'Collect Logs'), you could replay that command with Postman or curl/wget if you know how to do that.

My PAN integration just works with CPPM 6.10, I only have a single firewall and no Panorama. May be good to get PAN Support involved as well, as it may be Panorama acting strangely. Are you sending high numbers of userid updates (think multiple per second)?

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Aug 04, 2022 07:29 AM
From: Unknown User
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Curious why you have a new install of CPPM 6.5??? That version is well past support.  You should upgrade ClearPass to 6.10.
Original Message:
Sent: Aug 04, 2022 03:05 AM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hey Airheads,

I've got a new install of CPPM 6.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.

Then it looks like an issue in ClearPass. Please contact Aruba Support as I think to remember that I have seen once a similar issue reported which was 'known' and pending fix. It it is simple to upgrade to 6.10.6, you may give that a try before opening a TAC case.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
------------------------------

Original Message:
Sent: Aug 04, 2022 03:25 PM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

hey herman, thanks for the reply. 

it was just working fine but have also started seeing other errors in the cluster so wondering if something isn't happy internally. Looking at the palo we see successful login events from ClearPass so this could be intermittent. XML data is flowing but clearpass is reporting this error so thinking it could be a timeout type of scenario. there is a very large number of firewall serials in the request so wonder if thats related. 

Source PolicyServer
Level WARN
Category HTTP
Action Failed
Timestamp Aug 04, 2022 15:35:18 AEST
Description
Unable to communicate with HTTP server http://localhost:6179/async_netd/cmdctrl

Thinking these could be related. a restart of server cleared this for a while but it seems to have returned. 


Original Message:
Sent: Aug 04, 2022 10:17 AM
From: Herman Robers
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Title mentions CPPM 6.10, so may be a typo in the message.

Do you see something on the Panorama side in the logging? Have you create packet captures of the traffic between ClearPass and Panorama? Are certificates/trust for https configured and still valid?

If I freely interpret the error message, it looks like CPPM can connect to the Panorama, SSL sessions comes up (so probably certs are okay), then ClearPass sends the request, but never hears back something from Panorama. If you have the full URL, and can find the JSON/XML (think you can get that from the postauth.log file if you run a 'Collect Logs'), you could replay that command with Postman or curl/wget if you know how to do that.

My PAN integration just works with CPPM 6.10, I only have a single firewall and no Panorama. May be good to get PAN Support involved as well, as it may be Panorama acting strangely. Are you sending high numbers of userid updates (think multiple per second)?

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Aug 04, 2022 07:29 AM
From: Unknown User
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Curious why you have a new install of CPPM 6.5??? That version is well past support.  You should upgrade ClearPass to 6.10.
Original Message:
Sent: Aug 04, 2022 03:05 AM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hey Airheads,

I've got a new install of CPPM 6.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.

sorry was typo should be 6.10.5
Original Message:
Sent: Aug 04, 2022 07:29 AM
From: Unknown User
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Curious why you have a new install of CPPM 6.5??? That version is well past support.  You should upgrade ClearPass to 6.10.
Original Message:
Sent: Aug 04, 2022 03:05 AM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hey Airheads,

I've got a new install of CPPM 6.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.

Was there a resolution to this? I'm seeing the same thing on 6.10.4

------------------------------
ACCX #1239 || ACMX #1384 || ACEP || ACSP || CWNA || CWSP
------------------------------

Original Message:
Sent: Aug 04, 2022 03:15 PM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

sorry was typo should be 6.10.5
Original Message:
Sent: Aug 04, 2022 07:29 AM
From: Unknown User
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Curious why you have a new install of CPPM 6.5??? That version is well past support.  You should upgrade ClearPass to 6.10.
Original Message:
Sent: Aug 04, 2022 03:05 AM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hey Airheads,

I've got a new install of CPPM 6.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.

for me, it ended up being that some of the managed firewalls in panorama were down or disconnected. This resulted in the XML API call failing as from what i could tell.  when you use panorama, it basically proxies the request to each firewall directly through panorama and if any of the firewalls don't respond then the query times out. 

I ended up switching to a model where i pushed User-ID to 2 separate firewalls and then let user-id redistribution take care of sharing the data around. this was each enforcement profile was only dependent on 1 firewall not 50 of them!

hope that helps. since changing i've had no issues. 

Scott
Original Message:
Sent: Nov 03, 2022 11:07 AM
From: Zak Emerick
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Was there a resolution to this? I'm seeing the same thing on 6.10.4

------------------------------
ACCX #1239 || ACMX #1384 || ACEP || ACSP || CWNA || CWSP

Original Message:
Sent: Aug 04, 2022 03:15 PM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

sorry was typo should be 6.10.5
Original Message:
Sent: Aug 04, 2022 07:29 AM
From: Unknown User
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Curious why you have a new install of CPPM 6.5??? That version is well past support.  You should upgrade ClearPass to 6.10.
Original Message:
Sent: Aug 04, 2022 03:05 AM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hey Airheads,

I've got a new install of CPPM 6.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.

Hmm. That is unfortunate in my case. I just have one firewall and no Panorama. In the past it seems like 9.x code never had these issues. 

I'm going to switch to syslog filtering (not ideal) and see where I get there.

------------------------------
ACCX #1239 || ACMX #1384 || ACEP || ACSP || CWNA || CWSP
------------------------------

Original Message:
Sent: Nov 03, 2022 04:35 PM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

for me, it ended up being that some of the managed firewalls in panorama were down or disconnected. This resulted in the XML API call failing as from what i could tell.  when you use panorama, it basically proxies the request to each firewall directly through panorama and if any of the firewalls don't respond then the query times out.

I ended up switching to a model where i pushed User-ID to 2 separate firewalls and then let user-id redistribution take care of sharing the data around. this was each enforcement profile was only dependent on 1 firewall not 50 of them!

hope that helps. since changing i've had no issues.

Scott
Original Message:
Sent: Nov 03, 2022 11:07 AM
From: Zak Emerick
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Was there a resolution to this? I'm seeing the same thing on 6.10.4

------------------------------
ACCX #1239 || ACMX #1384 || ACEP || ACSP || CWNA || CWSP

Original Message:
Sent: Aug 04, 2022 03:15 PM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

sorry was typo should be 6.10.5
Original Message:
Sent: Aug 04, 2022 07:29 AM
From: Unknown User
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Curious why you have a new install of CPPM 6.5??? That version is well past support.  You should upgrade ClearPass to 6.10.
Original Message:
Sent: Aug 04, 2022 03:05 AM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hey Airheads,

I've got a new install of CPPM 6.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.

After re-engaging Palo support they have confirmed that this is a bug in PanOS starting in 10.2.x. 10.1.3 and below seem to not be affected. The fix is to upgrade to 10.2.3-h2 which is available, wait for 10.2.4 due to release next year, or downgrade 10.1.3.

------------------------------
ACCX #1239 || ACMX #1384 || ACEP || ACSP || CWNA || CWSP
------------------------------

Original Message:
Sent: Nov 04, 2022 09:17 AM
From: Zak Emerick
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hmm. That is unfortunate in my case. I just have one firewall and no Panorama. In the past it seems like 9.x code never had these issues. 

I'm going to switch to syslog filtering (not ideal) and see where I get there.

------------------------------
ACCX #1239 || ACMX #1384 || ACEP || ACSP || CWNA || CWSP

Original Message:
Sent: Nov 03, 2022 04:35 PM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

for me, it ended up being that some of the managed firewalls in panorama were down or disconnected. This resulted in the XML API call failing as from what i could tell.  when you use panorama, it basically proxies the request to each firewall directly through panorama and if any of the firewalls don't respond then the query times out.

I ended up switching to a model where i pushed User-ID to 2 separate firewalls and then let user-id redistribution take care of sharing the data around. this was each enforcement profile was only dependent on 1 firewall not 50 of them!

hope that helps. since changing i've had no issues.

Scott
Original Message:
Sent: Nov 03, 2022 11:07 AM
From: Zak Emerick
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Was there a resolution to this? I'm seeing the same thing on 6.10.4

------------------------------
ACCX #1239 || ACMX #1384 || ACEP || ACSP || CWNA || CWSP

Original Message:
Sent: Aug 04, 2022 03:15 PM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

sorry was typo should be 6.10.5
Original Message:
Sent: Aug 04, 2022 07:29 AM
From: Unknown User
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Curious why you have a new install of CPPM 6.5??? That version is well past support.  You should upgrade ClearPass to 6.10.
Original Message:
Sent: Aug 04, 2022 03:05 AM
From: Scott Doorey
Subject: CPPM 6.10 stopped working with PAN User-ID XML API

Hey Airheads,

I've got a new install of CPPM 6.5 integrated with Palo Alto Panorama using the XML API. 

Solution was tested out fine but after a few weeks started getting repeated error messages:

Unable to post request to PAN panorama.hostname, err: (HTTPSession): unable to execute POST request. err: Post https://panorama.hostname/api/?action=set&key=<KEY>&target=<SERIAL>&type=user-id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"


Anybody had this issue? 

TAC Case open but taking some time to align with the right engineers.