SD-WAN

Hi,

we are currently utilizing an Aruba 9004 Gateway cluster, managed via Aruba Central, at one of our branch offices.
The gateways are serving as the DNS servers for approx. 150-200 concurrent clients.

Recently, we have encountered frequent issues where DNS requests are not being answered. Upon investigation, it appears that packets are being policed/dropped on the gateway, likely due to an excessive number of packets/requests per second (pps).

#show datapath cp-bwm

Datapath CP Bandwidth Management Table Entries
-------------------------------------------
CPU  Contract     Rate      Policed    Avail Credits  Queued Bytes/Pkts
---  --------  ----------  ----------  -------------  -----------------
1    4097            320 pps           0             10         0/0
1    4098           8000 pps           0            250         0/0
1    4099           2016 pps           0             63         0/0
1    4100           2016 pps           0             63         0/0
1    4101           8000 pps     2162688            250         0/0
1    4102           1024 pps           0             32         0/0
1    4103            128 pps  3935371267             -1     10890/125      <-----------------------
1    4104           1024 pps           0             32         0/0
1    4105            512 pps           0             16         0/0

As a temporary workaround, we are currently routing DNS requests to our server located in the datacenter. However, we are seeking recommendations on troubleshooting steps or specific commands that can be used on the Aruba 9004 Gateway to identify and address the root cause of the issue.

We have adjusted the DNS cache size on the gateway (via Central), setting it to the max value of 4096, as we determinded that the default cache size of 150 was probably insufficient for the number of clients. Unfortunately, increasing the DNS cache size to 4096 has not mitigated the issue.

I would appreciate your assistance in resolving this matter.

Thanks!
Mario

For these issues please open a case with TAC as you have 24x7 TAC support .

in the mean time what firmware version are yo running on the BGWs?

Hi,

we are currently utilizing an Aruba 9004 Gateway cluster, managed via Aruba Central, at one of our branch offices.
The gateways are serving as the DNS servers for approx. 150-200 concurrent clients.

Recently, we have encountered frequent issues where DNS requests are not being answered. Upon investigation, it appears that packets are being policed/dropped on the gateway, likely due to an excessive number of packets/requests per second (pps).

#show datapath cp-bwm

Datapath CP Bandwidth Management Table Entries
-------------------------------------------
CPU  Contract     Rate      Policed    Avail Credits  Queued Bytes/Pkts
---  --------  ----------  ----------  -------------  -----------------
1    4097            320 pps           0             10         0/0
1    4098           8000 pps           0            250         0/0
1    4099           2016 pps           0             63         0/0
1    4100           2016 pps           0             63         0/0
1    4101           8000 pps     2162688            250         0/0
1    4102           1024 pps           0             32         0/0
1    4103            128 pps  3935371267             -1     10890/125      <-----------------------
1    4104           1024 pps           0             32         0/0
1    4105            512 pps           0             16         0/0

As a temporary workaround, we are currently routing DNS requests to our server located in the datacenter. However, we are seeking recommendations on troubleshooting steps or specific commands that can be used on the Aruba 9004 Gateway to identify and address the root cause of the issue.

We have adjusted the DNS cache size on the gateway (via Central), setting it to the max value of 4096, as we determinded that the default cache size of 150 was probably insufficient for the number of clients. Unfortunately, increasing the DNS cache size to 4096 has not mitigated the issue.

I would appreciate your assistance in resolving this matter.

Thanks!
Mario

Hi ariyap

Thanks for your reply.

For these issues please open a case with TAC as you have 24x7 TAC support .

in the mean time what firmware version are yo running on the BGWs?

Hi,

we are currently utilizing an Aruba 9004 Gateway cluster, managed via Aruba Central, at one of our branch offices.
The gateways are serving as the DNS servers for approx. 150-200 concurrent clients.

Recently, we have encountered frequent issues where DNS requests are not being answered. Upon investigation, it appears that packets are being policed/dropped on the gateway, likely due to an excessive number of packets/requests per second (pps).

#show datapath cp-bwm

Datapath CP Bandwidth Management Table Entries
-------------------------------------------
CPU  Contract     Rate      Policed    Avail Credits  Queued Bytes/Pkts
---  --------  ----------  ----------  -------------  -----------------
1    4097            320 pps           0             10         0/0
1    4098           8000 pps           0            250         0/0
1    4099           2016 pps           0             63         0/0
1    4100           2016 pps           0             63         0/0
1    4101           8000 pps     2162688            250         0/0
1    4102           1024 pps           0             32         0/0
1    4103            128 pps  3935371267             -1     10890/125      <-----------------------
1    4104           1024 pps           0             32         0/0
1    4105            512 pps           0             16         0/0

As a temporary workaround, we are currently routing DNS requests to our server located in the datacenter. However, we are seeking recommendations on troubleshooting steps or specific commands that can be used on the Aruba 9004 Gateway to identify and address the root cause of the issue.

We have adjusted the DNS cache size on the gateway (via Central), setting it to the max value of 4096, as we determinded that the default cache size of 150 was probably insufficient for the number of clients. Unfortunately, increasing the DNS cache size to 4096 has not mitigated the issue.

I would appreciate your assistance in resolving this matter.

Thanks!
Mario

I could not find any DNS cache size  related fixes. But still I suggest to upgrade to 10.6.0.3 as there were Multiple  Vulnerabilities reported a few weeks ago that is fixed in 10.6.0.3

Please open a TAC case for your DNS cache size issue.

Hi ariyap

Thanks for your reply.

For these issues please open a case with TAC as you have 24x7 TAC support .

in the mean time what firmware version are yo running on the BGWs?

Hi,

we are currently utilizing an Aruba 9004 Gateway cluster, managed via Aruba Central, at one of our branch offices.
The gateways are serving as the DNS servers for approx. 150-200 concurrent clients.

Recently, we have encountered frequent issues where DNS requests are not being answered. Upon investigation, it appears that packets are being policed/dropped on the gateway, likely due to an excessive number of packets/requests per second (pps).

#show datapath cp-bwm

Datapath CP Bandwidth Management Table Entries
-------------------------------------------
CPU  Contract     Rate      Policed    Avail Credits  Queued Bytes/Pkts
---  --------  ----------  ----------  -------------  -----------------
1    4097            320 pps           0             10         0/0
1    4098           8000 pps           0            250         0/0
1    4099           2016 pps           0             63         0/0
1    4100           2016 pps           0             63         0/0
1    4101           8000 pps     2162688            250         0/0
1    4102           1024 pps           0             32         0/0
1    4103            128 pps  3935371267             -1     10890/125      <-----------------------
1    4104           1024 pps           0             32         0/0
1    4105            512 pps           0             16         0/0

As a temporary workaround, we are currently routing DNS requests to our server located in the datacenter. However, we are seeking recommendations on troubleshooting steps or specific commands that can be used on the Aruba 9004 Gateway to identify and address the root cause of the issue.

We have adjusted the DNS cache size on the gateway (via Central), setting it to the max value of 4096, as we determinded that the default cache size of 150 was probably insufficient for the number of clients. Unfortunately, increasing the DNS cache size to 4096 has not mitigated the issue.

I would appreciate your assistance in resolving this matter.

Thanks!
Mario

We already had a TAC case opened for this. But we will try again.

Message from TAC, March 2023:

"

When the issue happens (DNS failures) We see packets being dropped on the BGW due to an limitation on the controller on the current code (supported DNS-Cache entries are 150).

There is an CLI to increase this number but the target is 10.5. (Configurable upto 10K)

Currently we do not have any workaround apart from restarting the DNS service.

So the recommendation is to "not to use BGW as an DNS server" until 10.5.

Also the target for 10.5 is not decided yet, If any updates I would let you know.

"

In Sept 2023 10.5 got released. We waited with the update because we wanted to stay on the LSR and it did not get released on LSR sadly.

Now we're on 10.6, set the cache to the max possible value (4096) but still face the same issue. We will try to find out what's generating that many requests but as we had the same behaviour on two different sites, it's more likely we're hitting a bug/dns server config issue here...

If you have any recommendation to go forward with debugging that issue, please let us know.

Thanks for your help!

Original Message:
Sent: Oct 04, 2024 02:36 AM
From: ariyap
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

I could not find any DNS cache size  related fixes. But still I suggest to upgrade to 10.6.0.3 as there were Multiple  Vulnerabilities reported a few weeks ago that is fixed in 10.6.0.3

Please open a TAC case for your DNS cache size issue.

------------------------------
If my post was useful accept solution and/or give kudos.
Any opinions expressed here are solely my own and not necessarily that of HPE or Aruba.

Original Message:
Sent: Oct 04, 2024 02:04 AM
From: ML61
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

Hi ariyap

Thanks for your reply.

Original Message:
Sent: Oct 03, 2024 07:48 PM
From: ariyap
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

For these issues please open a case with TAC as you have 24x7 TAC support .

in the mean time what firmware version are yo running on the BGWs?

------------------------------
If my post was useful accept solution and/or give kudos.
Any opinions expressed here are solely my own and not necessarily that of HPE or Aruba.

Original Message:
Sent: Oct 03, 2024 10:21 AM
From: ML61
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

Hi,

we are currently utilizing an Aruba 9004 Gateway cluster, managed via Aruba Central, at one of our branch offices.
The gateways are serving as the DNS servers for approx. 150-200 concurrent clients.

Recently, we have encountered frequent issues where DNS requests are not being answered. Upon investigation, it appears that packets are being policed/dropped on the gateway, likely due to an excessive number of packets/requests per second (pps).

#show datapath cp-bwm

Datapath CP Bandwidth Management Table Entries
-------------------------------------------
CPU  Contract     Rate      Policed    Avail Credits  Queued Bytes/Pkts
---  --------  ----------  ----------  -------------  -----------------
1    4097            320 pps           0             10         0/0
1    4098           8000 pps           0            250         0/0
1    4099           2016 pps           0             63         0/0
1    4100           2016 pps           0             63         0/0
1    4101           8000 pps     2162688            250         0/0
1    4102           1024 pps           0             32         0/0
1    4103            128 pps  3935371267             -1     10890/125      <-----------------------
1    4104           1024 pps           0             32         0/0
1    4105            512 pps           0             16         0/0

As a temporary workaround, we are currently routing DNS requests to our server located in the datacenter. However, we are seeking recommendations on troubleshooting steps or specific commands that can be used on the Aruba 9004 Gateway to identify and address the root cause of the issue.

We have adjusted the DNS cache size on the gateway (via Central), setting it to the max value of 4096, as we determinded that the default cache size of 150 was probably insufficient for the number of clients. Unfortunately, increasing the DNS cache size to 4096 has not mitigated the issue.

I would appreciate your assistance in resolving this matter.

Thanks!
Mario

From what you describe, it appears that your clients exceed the allowed Control-Plane limit of the number requests (pps). The CP bandwidth contracts protect the gateway from too many request (sort of a DDoS protection).

Increasing the cache size will not mitigate your issue as the gateway still has to respond to every DNS query sent by your clients. Caching only prevents the gateway from redirecting a DNS request to an upstream server, in order to resolve a query for which it doesn't have a local entry.

As long as your clients keep querying the gateway at a high rate, it will still lead to requests being policed (dropped by CP bandwidth-contracts). Only true resolution would be to investigate why there is such a high number of requests being sent to the gateway in the first place.

Increasing CP thresholds is something that is investigated but always weighed very carefully, as there are finite resources in the operating system.

------------------------------
I work for Aruba. Any opinions expressed here are solely my own and not do not represent that of Hewlett Packard Enterprise or Aruba.

Original Message:
Sent: Oct 04, 2024 03:45 AM
From: ML61
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

We already had a TAC case opened for this. But we will try again.

Message from TAC, March 2023:

"

When the issue happens (DNS failures) We see packets being dropped on the BGW due to an limitation on the controller on the current code (supported DNS-Cache entries are 150).

There is an CLI to increase this number but the target is 10.5. (Configurable upto 10K)

Currently we do not have any workaround apart from restarting the DNS service.

So the recommendation is to "not to use BGW as an DNS server" until 10.5.

Also the target for 10.5 is not decided yet, If any updates I would let you know.

"

In Sept 2023 10.5 got released. We waited with the update because we wanted to stay on the LSR and it did not get released on LSR sadly.

Now we're on 10.6, set the cache to the max possible value (4096) but still face the same issue. We will try to find out what's generating that many requests but as we had the same behaviour on two different sites, it's more likely we're hitting a bug/dns server config issue here...

If you have any recommendation to go forward with debugging that issue, please let us know.

Thanks for your help!

Original Message:
Sent: Oct 04, 2024 02:36 AM
From: ariyap
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

I could not find any DNS cache size  related fixes. But still I suggest to upgrade to 10.6.0.3 as there were Multiple  Vulnerabilities reported a few weeks ago that is fixed in 10.6.0.3

Please open a TAC case for your DNS cache size issue.

------------------------------
If my post was useful accept solution and/or give kudos.
Any opinions expressed here are solely my own and not necessarily that of HPE or Aruba.

Original Message:
Sent: Oct 04, 2024 02:04 AM
From: ML61
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

Hi ariyap

Thanks for your reply.

Original Message:
Sent: Oct 03, 2024 07:48 PM
From: ariyap
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

For these issues please open a case with TAC as you have 24x7 TAC support .

in the mean time what firmware version are yo running on the BGWs?

------------------------------
If my post was useful accept solution and/or give kudos.
Any opinions expressed here are solely my own and not necessarily that of HPE or Aruba.

Original Message:
Sent: Oct 03, 2024 10:21 AM
From: ML61
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

Hi,

we are currently utilizing an Aruba 9004 Gateway cluster, managed via Aruba Central, at one of our branch offices.
The gateways are serving as the DNS servers for approx. 150-200 concurrent clients.

Recently, we have encountered frequent issues where DNS requests are not being answered. Upon investigation, it appears that packets are being policed/dropped on the gateway, likely due to an excessive number of packets/requests per second (pps).

#show datapath cp-bwm

Datapath CP Bandwidth Management Table Entries
-------------------------------------------
CPU  Contract     Rate      Policed    Avail Credits  Queued Bytes/Pkts
---  --------  ----------  ----------  -------------  -----------------
1    4097            320 pps           0             10         0/0
1    4098           8000 pps           0            250         0/0
1    4099           2016 pps           0             63         0/0
1    4100           2016 pps           0             63         0/0
1    4101           8000 pps     2162688            250         0/0
1    4102           1024 pps           0             32         0/0
1    4103            128 pps  3935371267             -1     10890/125      <-----------------------
1    4104           1024 pps           0             32         0/0
1    4105            512 pps           0             16         0/0

As a temporary workaround, we are currently routing DNS requests to our server located in the datacenter. However, we are seeking recommendations on troubleshooting steps or specific commands that can be used on the Aruba 9004 Gateway to identify and address the root cause of the issue.

We have adjusted the DNS cache size on the gateway (via Central), setting it to the max value of 4096, as we determinded that the default cache size of 150 was probably insufficient for the number of clients. Unfortunately, increasing the DNS cache size to 4096 has not mitigated the issue.

I would appreciate your assistance in resolving this matter.

Thanks!
Mario

Hi Oliver

Thanks a lot for your detailed response!

I understand that the Control Processor Bandwidth Contract traffic rate for DNS is limited to 128 packets per second on the Aruba Gateway 9004, likely as a measure to maintain optimal performance within the gateway's processing capabilities.

cp-bandwidth-contract cpbwc-ipv4-dns pps 128
firewall cp
 ipv4 permit any proto 17 ports 53 53 bandwidth-contract cpbwc-ipv4-dns

Would you be able to provide information on the Control Processor Bandwidth Contract traffic rate set on the Aruba Gateway 9114?

If maintaining a DNS traffic rate below 128 packets per second isn't feasible for our needs, what options would you recommend?

Thanks!

Mario

Original Message:
Sent: Nov 05, 2024 09:24 AM
From: Oliver Wehrli
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

From what you describe, it appears that your clients exceed the allowed Control-Plane limit of the number requests (pps). The CP bandwidth contracts protect the gateway from too many request (sort of a DDoS protection).

Increasing the cache size will not mitigate your issue as the gateway still has to respond to every DNS query sent by your clients. Caching only prevents the gateway from redirecting a DNS request to an upstream server, in order to resolve a query for which it doesn't have a local entry.

As long as your clients keep querying the gateway at a high rate, it will still lead to requests being policed (dropped by CP bandwidth-contracts). Only true resolution would be to investigate why there is such a high number of requests being sent to the gateway in the first place.

Increasing CP thresholds is something that is investigated but always weighed very carefully, as there are finite resources in the operating system.

------------------------------
I work for Aruba. Any opinions expressed here are solely my own and not do not represent that of Hewlett Packard Enterprise or Aruba.

Original Message:
Sent: Oct 04, 2024 03:45 AM
From: ML61
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

We already had a TAC case opened for this. But we will try again.

Message from TAC, March 2023:

"

When the issue happens (DNS failures) We see packets being dropped on the BGW due to an limitation on the controller on the current code (supported DNS-Cache entries are 150).

There is an CLI to increase this number but the target is 10.5. (Configurable upto 10K)

Currently we do not have any workaround apart from restarting the DNS service.

So the recommendation is to "not to use BGW as an DNS server" until 10.5.

Also the target for 10.5 is not decided yet, If any updates I would let you know.

"

In Sept 2023 10.5 got released. We waited with the update because we wanted to stay on the LSR and it did not get released on LSR sadly.

Now we're on 10.6, set the cache to the max possible value (4096) but still face the same issue. We will try to find out what's generating that many requests but as we had the same behaviour on two different sites, it's more likely we're hitting a bug/dns server config issue here...

If you have any recommendation to go forward with debugging that issue, please let us know.

Thanks for your help!

Original Message:
Sent: Oct 04, 2024 02:36 AM
From: ariyap
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

I could not find any DNS cache size  related fixes. But still I suggest to upgrade to 10.6.0.3 as there were Multiple  Vulnerabilities reported a few weeks ago that is fixed in 10.6.0.3

Please open a TAC case for your DNS cache size issue.

------------------------------
If my post was useful accept solution and/or give kudos.
Any opinions expressed here are solely my own and not necessarily that of HPE or Aruba.

Original Message:
Sent: Oct 04, 2024 02:04 AM
From: ML61
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

Hi ariyap

Thanks for your reply.

Original Message:
Sent: Oct 03, 2024 07:48 PM
From: ariyap
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

For these issues please open a case with TAC as you have 24x7 TAC support .

in the mean time what firmware version are yo running on the BGWs?

------------------------------
If my post was useful accept solution and/or give kudos.
Any opinions expressed here are solely my own and not necessarily that of HPE or Aruba.

Original Message:
Sent: Oct 03, 2024 10:21 AM
From: ML61
Subject: Aruba 9004 Gateway Cluster - DNS Request Handling Issues

Hi,

we are currently utilizing an Aruba 9004 Gateway cluster, managed via Aruba Central, at one of our branch offices.
The gateways are serving as the DNS servers for approx. 150-200 concurrent clients.

Recently, we have encountered frequent issues where DNS requests are not being answered. Upon investigation, it appears that packets are being policed/dropped on the gateway, likely due to an excessive number of packets/requests per second (pps).

#show datapath cp-bwm

Datapath CP Bandwidth Management Table Entries
-------------------------------------------
CPU  Contract     Rate      Policed    Avail Credits  Queued Bytes/Pkts
---  --------  ----------  ----------  -------------  -----------------
1    4097            320 pps           0             10         0/0
1    4098           8000 pps           0            250         0/0
1    4099           2016 pps           0             63         0/0
1    4100           2016 pps           0             63         0/0
1    4101           8000 pps     2162688            250         0/0
1    4102           1024 pps           0             32         0/0
1    4103            128 pps  3935371267             -1     10890/125      <-----------------------
1    4104           1024 pps           0             32         0/0
1    4105            512 pps           0             16         0/0

As a temporary workaround, we are currently routing DNS requests to our server located in the datacenter. However, we are seeking recommendations on troubleshooting steps or specific commands that can be used on the Aruba 9004 Gateway to identify and address the root cause of the issue.

We have adjusted the DNS cache size on the gateway (via Central), setting it to the max value of 4096, as we determinded that the default cache size of 150 was probably insufficient for the number of clients. Unfortunately, increasing the DNS cache size to 4096 has not mitigated the issue.

I would appreciate your assistance in resolving this matter.

Thanks!
Mario