Wireless Access

Hi everyone

I'm inquiry about Controller log Message.

I could not find about log message.

Controller Info as follows :

1. Aruba : 3400.

2. Aruba OS : 6.3.1.13

3. Aruba licensed : Access Point : 64 PEF : 64

4. IP assignment to User by DHCP server of Aruba Controller.

log Message as follows :

" show log system 100 "

Jan 20 13:54:24 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458217. DNS max IP to name limit reached android.clients.google.com with 8877 num_ips
Jan 20 13:54:58 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458219. DNS max IP to name limit reached android.clients.google.com with 8877 num_ips
Jan 20 13:55:48 :334302:  <ERRS> |FW Visibility|  fw_dest_new_ip_node: MM: 458222. DNS max IP to name limit reached android.clients.google.com with 8877 num_ips
Jan 20 13:55:48 :334302:  <ERRS> |FW Visibility|  fw_dest_new_ip_node: MM: 458222. DNS max IP to name limit reached android.clients.google.com with 8878 num_ips
Jan 20 13:55:48 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458222. DNS max IP to name limit reached android.clients.google.com with 8879 num_ips
Jan 20 13:56:22 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458224. DNS max IP to name limit reached android.clients.google.com with 8879 num_ips
Jan 20 13:56:38 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458225. DNS max IP to name limit reached android.clients.google.com with 8879 num_ips
Jan 20 13:57:12 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458227. DNS max IP to name limit reached android.clients.google.com with 8879 num_ips
Jan 20 13:57:29 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458228. DNS max IP to name limit reached android.clients.google.com with 8884 num_ips
Jan 20 13:57:29 :334302:  <ERRS> |FW Visibility|  fw_dest_new_ip_node: MM: 458228. DNS max IP to name limit reached mtalk.google.com with 2048 num_ips
Jan 20 13:58:02 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458230. DNS max IP to name limit reached android.clients.google.com with 8884 num_ips
Jan 20 13:58:19 :334302:  <ERRS> |FW Visibility|  fw_dest_new_ip_node: MM: 458231. DNS max IP to name limit reached android.clients.google.com with 8884 num_ips
Jan 20 13:58:36 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458232. DNS max IP to name limit reached android.clients.google.com with 8884 num_ips
Jan 20 13:58:53 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458233. DNS max IP to name limit reached android.clients.google.com with 8884 num_ips
Jan 20 13:59:26 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458235. DNS max IP to name limit reached android.clients.google.com with 8884 num_ips
Jan 20 13:59:26 :334302:  <ERRS> |FW Visibility|  fw_dest_new_ip_node: MM: 458235. DNS max IP to name limit reached android.clients.google.com with 8884 num_ips
Jan 20 13:59:26 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458235. DNS max IP to name limit reached android.clients.google.com with 8885 num_ips
Jan 20 14:00:16 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458238. DNS max IP to name limit reached android.clients.google.com with 8885 num_ips
Jan 20 14:00:33 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458239. DNS max IP to name limit reached nlp.samsungcic.com with 2055 num_ips
Jan 20 14:00:33 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458239. DNS max IP to name limit reached android.clients.google.com with 8885 num_ips
Jan 20 14:01:24 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458242. DNS max IP to name limit reached android.clients.google.com with 8885 num_ips
Jan 20 14:02:14 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 458245. DNS max IP to name limit reached android.clients.google.com with 8885 num_ips
Jan 20 14:02:47 :334302:  <ERRS> |FW Visibility|  fw_dest_new_ip_node: MM: 458247. DNS max IP to name limit reached android.clients.google.com with 8885 num_ips
Jan 20 14:02:47 :334302:  <ERRS> |FW Visibility|  fw_dest_new_ip_node: MM: 458247. DNS max IP to name limit reached android.clients.google.com with 8888 num_ips

this log message is what is problem? attached is log message file

Thank you

regards

IJ

Attachment(s)

Aruba Conctorller Syslog Message.txt   15 KB 1 version

This error is often seen when DNS names resolve to numerous IP addresses. What issue are you trying to fix?

Currently.

I was not action about this issue.

and I would like know to method of resolve. 

Regards

IJ

This issue was resolved in ArubaOS 6.4.2.3 with bug 108533 in the release notes here:  http://support.arubanetworks.com/Documentation/tabid/77/DMXModule/512/Command/Core_Download/Default.aspx?EntryId=16112

OK Thank you for answer to me

Thank you.!!!

Regards

IJ

Hello,

Is there a fix for this error under Aruba 6.3 OS? It has rendered the logs on my controllers unusable.

Thank you

It has only been reported/fixed in 6.4.x.

Please open a TAC case if you are observing this on other versions of ArubaOS code.

it's also fixed in 6.3.1.14 (November 2014) - no need to re-raise it

It appears to be still happening in 6.3.1.16 albeit with higher limits. Do you know what the limit was raised to in 6.3? Is there a way to increase the value from the admin side?

There may be a correlation with FW Visibility events and CPU spikes. Still analyzing logs. Sampling of FW Visibility events below. Happening on more than one local, all M3k.

Nov 27 14:27:16 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 1209183. DNS max IP to name limit reached clients4.google.com with 53227 num_ips
Nov 28 02:33:53 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 1211787. DNS max IP to name limit reached safebrowsing.google.com with 53490 num_ips
Nov 28 04:54:52 :334302:  <ERRS> |FW Visibility|  fw_rcv_dns_export: MM: 1212292. FW Visiblity DNS DMAed buffer out of bounds (entries 180)
Nov 28 08:00:44 :334302:  <ERRS> |FW Visibility|  fw_rcv_dns_export: MM: 1212958. FW Visiblity DNS DMAed buffer out of bounds (entries 232)
Nov 28 08:12:42 :334302:  <ERRS> |FW Visibility|  fw_rcv_dns_export: MM: 1213001. FW Visiblity DNS DMAed buffer out of bounds (entries 224)
Nov 28 17:54:25 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 1215086. DNS max IP to name limit reached 1-courier.push.apple.com with 49376 num_ips
Nov 29 05:17:13 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 1217533. DNS max IP to name limit reached www.google.com with 38508 num_ips
Nov 29 15:12:26 :334302:  <ERRS> |FW Visibility|  fw_dest_ip_name_remap: MM: 1219666. DNS max IP to name limit reached www.google.com with 38575 num_ips
Nov 29 20:32:20  KERNEL: 1:alloc_pages : Current task : fw_visibility, order : 0
Nov 29 20:32:20  KERNEL: 1:alloc_pages : Current task : fw_visibility, order : 0
Nov 29 20:32:20  KERNEL: 1:alloc_pages : Current task : fw_visibility, order : 0
Nov 29 20:32:20  KERNEL: 1:alloc_pages : Current task : fw_visibility, order : 0

Mike

hi Mike

the limit was meant to be 32k, and the log frequency reduced. There is no way to increase/change it. Having said that, the below log is implying > 32k (in one case 53490 ips) - was that on 6.3.1.16 ?

the other messages (err level) do suggest that there is memory issues occurring as a result of the fw visibility.

regards

-jeff

Thanks Jeff. Yes it appears we have a few busy controllers in the res halls!

I have a couple of questions:

1. Is this user impacting?

2. Is it preferred to disable FW visibility for now?

3. Is there a way to shim in a DNS caching server or have the controller cache the queries?

Thanks,

MIke

hi Mike

to your q's above...

1. no, this is just part of fw vis

2. probably (if you dont use it, then yes just disable it) - but before you do, please send me a tar logs techsup and i will raise a new bug on it for you if you like (can you confirm the line showing 53490 was on 6.3.1.16 ?). If the controller has plenty of free memory then this should mostly be a cosmetic issue that would only cause issues if you were using the fw vis dashboard or data.

3. this is only for the fw vis stuff, it doesnt touch the client DNS packets at all.

I dont mind to raise a ticket / bug for you after reviewing the tech sup (or you can use whatever path you usually take for opening a ticket with Aruba). My email is my forum name at arubanetworks dot com.

regards

-jeff

Thanks Jeff. I'll send you the tar logs techsup as soon as I can.

I have one other question. Is this really hammering away at our DNS server? Seems like an awful lot of DNS queries..

Dec  8 00:58:06  FW Visibility[2068]: <334302> <ERRS> < xxx.xxx.xx.24>  fw_dest_ip_name_remap: MM: 404171. DNS max IP to name limit reached clients4.google.com with 68723 num_ips
Dec  8 04:18:50  FW Visibility[2100]: <334302> <ERRS> < xxx.xxx.xx.25>  fw_dest_ip_name_remap: MM: 1263756. DNS max IP to name limit reached android.clients.google.com with 36248 num_ips
Dec  8 07:02:50  FW Visibility[2068]: <334302> <ERRS> < xxx.xxx.xx.24>  fw_dest_ip_name_remap: MM: 405478. DNS max IP to name limit reached api.mixpanel.com with 33519 num_ips
Dec  8 08:22:45  FW Visibility[32346]: <334302> <ERRS> < xxx.xxx.xx.30>  fw_dest_ip_name_remap: MM: 1264675. DNS max IP to name limit reached www.google.com with 35682 num_ips
Dec  8 08:47:27  FW Visibility[2076]: <334302> <ERRS> < xxx.xxx.xx.26>  fw_dest_ip_name_remap: MM: 366012. DNS max IP to name limit reached clients4.google.com with 62062 num_ips

-Mike

clients are hammering away on DNS irrespective, all this is saying is that (apparently) there are some 60k ip address that have been returned for (say) clients4.google.com and thats "too many"

I say apparently because when the bug for this first came up, I put it to R&D that this seems way out there, it is highly unlikely that "api.mixpanel.com" has access to this many IP addresses . I ran constant DNS queries for a few usual suspects for several hours from various geographies and summed up the unique ip addresses seen. Whist there are a lot, its orders of magnitude smaller than 32k or 64k.

To me, something is leaking. Let me get in touch with the R&D and see what I can find out.

** edit to my reply **  bug 114189 already exists for this, it's not fixed yet - QA was able to reproduce. Will try to hurry up R&D for an update/plan to resolve. In the mean time, you can consider disabling fw_vis - especially if the controller is starting to run low on memory (when you send tar logs techsupport I can check into that closer and advise)

regards

-jeff