Wireless Access

Hello everyone,

I posted about this over the Summer of 2024 but never got anywhere with TAC, and the symptoms stopped once the students arrived back to campus, so I thought I'd reach out again for a fresh look. With the assistance of our Aruba SE, we upgraded our core switch from an HP 8212zl to an Aruba 6405 back in June 2024. After about a day, we noticed that our NAC (FortiNAC) was intermittently failing SNMP polls and pings to our two Aruba 7210 wireless controllers. If we check the controllers, they never went down and neither did any of the APs. The same goes for the core switch. There's no evidence of packet loss on the ports and the ports never lost connection to the core.

Our core is on VLAN 1 and our Aruba controllers are on VLAN 40 with no ACLs or firewall in between. After spending a lot of time looking at logs on the wireless side, I had noticed that our mobility manager, that manages the two controller cluster, would drop connection to each controller every 51 minutes, at different times in the hour. After 10 seconds, the connection would re-establish. Our mobility master was also on VLAN 1, not on VLAN 40 with the controllers.

Thinking that the new core is doing something different than the old HP, we went down that rabbit hole but didn't come up with anything significant. However, I did look back at a TAC case from 2023 where we pulled tech support logs, and sure enough, the MM was dropping connection to both controllers at the same rate of every 51 minutes and we didn't know it. And this was happening on the old core switch so that rules out the 6405 causing the issue.

With the help of our Aruba SE, we moved the MM to VLAN 40 and the 51 min connection drop between the MM and the controllers has stopped. Now FortiNAC had continued to alert us to SNMP poll failures (contact lost) up until students started arriving back on campus in late August. At that point, the FortiNAC alerts stopped and everything was normal again. Well that changed last week when all of the students left campus for Thanksgiving break and the issue has returned.

What could possibly be the issue here? We are running an old version of Aruba OS (8.7.1.11) on the MM and MDs and I thought that an upgrade may help. However, the tech support logs taken back in the Summer of 2023 were taken when we dipped out toes into AOS 8.10 but had to downgrade due to some major AirGroup issues. So the issue occurred on AOS 8.7 and 8.10, and on both our old HP 8212zl core as well as our new Aruba 6405 core. The NAC and all wifi MM and MDs are connected directly to the core via 10GB fiber.

Could there be packet loss inbetween the VLANs? Could the controllers be dropping packets intermittently? 

Any ideas? I'm desperate at this point.

Can't say that I have a specific recommendation to make on this, but every time I see someone using VLAN 1 in production for anything there seems to be problems associated with doing so.  Too many devices treat VLAN 1 differently from other VLANs for me to ever trust using that VLAN for anything other than the native VLAN when running MSTP.

------------------------------
Carson Hulcher, ACEX#110

Original Message:
Sent: Nov 29, 2024 10:17 AM
From: nkuhl30
Subject: Possible inter-VLAN packet loss

Hello everyone,

I posted about this over the Summer of 2024 but never got anywhere with TAC, and the symptoms stopped once the students arrived back to campus, so I thought I'd reach out again for a fresh look. With the assistance of our Aruba SE, we upgraded our core switch from an HP 8212zl to an Aruba 6405 back in June 2024. After about a day, we noticed that our NAC (FortiNAC) was intermittently failing SNMP polls and pings to our two Aruba 7210 wireless controllers. If we check the controllers, they never went down and neither did any of the APs. The same goes for the core switch. There's no evidence of packet loss on the ports and the ports never lost connection to the core.

Our core is on VLAN 1 and our Aruba controllers are on VLAN 40 with no ACLs or firewall in between. After spending a lot of time looking at logs on the wireless side, I had noticed that our mobility manager, that manages the two controller cluster, would drop connection to each controller every 51 minutes, at different times in the hour. After 10 seconds, the connection would re-establish. Our mobility master was also on VLAN 1, not on VLAN 40 with the controllers.

Thinking that the new core is doing something different than the old HP, we went down that rabbit hole but didn't come up with anything significant. However, I did look back at a TAC case from 2023 where we pulled tech support logs, and sure enough, the MM was dropping connection to both controllers at the same rate of every 51 minutes and we didn't know it. And this was happening on the old core switch so that rules out the 6405 causing the issue.

With the help of our Aruba SE, we moved the MM to VLAN 40 and the 51 min connection drop between the MM and the controllers has stopped. Now FortiNAC had continued to alert us to SNMP poll failures (contact lost) up until students started arriving back on campus in late August. At that point, the FortiNAC alerts stopped and everything was normal again. Well that changed last week when all of the students left campus for Thanksgiving break and the issue has returned.

What could possibly be the issue here? We are running an old version of Aruba OS (8.7.1.11) on the MM and MDs and I thought that an upgrade may help. However, the tech support logs taken back in the Summer of 2023 were taken when we dipped out toes into AOS 8.10 but had to downgrade due to some major AirGroup issues. So the issue occurred on AOS 8.7 and 8.10, and on both our old HP 8212zl core as well as our new Aruba 6405 core. The NAC and all wifi MM and MDs are connected directly to the core via 10GB fiber.

Could there be packet loss inbetween the VLANs? Could the controllers be dropping packets intermittently? 

Any ideas? I'm desperate at this point.

Thanks. This is helpful. We have our core and all of our edge switches on VLAN 1, along with our NAC. So is it your recommendation is to only have switches on VLAN 1 and keep other servers off of it?

Original Message:
Sent: Dec 02, 2024 10:25 AM
From: chulcher
Subject: Possible inter-VLAN packet loss

Can't say that I have a specific recommendation to make on this, but every time I see someone using VLAN 1 in production for anything there seems to be problems associated with doing so.  Too many devices treat VLAN 1 differently from other VLANs for me to ever trust using that VLAN for anything other than the native VLAN when running MSTP.

------------------------------
Carson Hulcher, ACEX#110

Original Message:
Sent: Nov 29, 2024 10:17 AM
From: nkuhl30
Subject: Possible inter-VLAN packet loss

Hello everyone,

I posted about this over the Summer of 2024 but never got anywhere with TAC, and the symptoms stopped once the students arrived back to campus, so I thought I'd reach out again for a fresh look. With the assistance of our Aruba SE, we upgraded our core switch from an HP 8212zl to an Aruba 6405 back in June 2024. After about a day, we noticed that our NAC (FortiNAC) was intermittently failing SNMP polls and pings to our two Aruba 7210 wireless controllers. If we check the controllers, they never went down and neither did any of the APs. The same goes for the core switch. There's no evidence of packet loss on the ports and the ports never lost connection to the core.

Our core is on VLAN 1 and our Aruba controllers are on VLAN 40 with no ACLs or firewall in between. After spending a lot of time looking at logs on the wireless side, I had noticed that our mobility manager, that manages the two controller cluster, would drop connection to each controller every 51 minutes, at different times in the hour. After 10 seconds, the connection would re-establish. Our mobility master was also on VLAN 1, not on VLAN 40 with the controllers.

Thinking that the new core is doing something different than the old HP, we went down that rabbit hole but didn't come up with anything significant. However, I did look back at a TAC case from 2023 where we pulled tech support logs, and sure enough, the MM was dropping connection to both controllers at the same rate of every 51 minutes and we didn't know it. And this was happening on the old core switch so that rules out the 6405 causing the issue.

With the help of our Aruba SE, we moved the MM to VLAN 40 and the 51 min connection drop between the MM and the controllers has stopped. Now FortiNAC had continued to alert us to SNMP poll failures (contact lost) up until students started arriving back on campus in late August. At that point, the FortiNAC alerts stopped and everything was normal again. Well that changed last week when all of the students left campus for Thanksgiving break and the issue has returned.

What could possibly be the issue here? We are running an old version of Aruba OS (8.7.1.11) on the MM and MDs and I thought that an upgrade may help. However, the tech support logs taken back in the Summer of 2023 were taken when we dipped out toes into AOS 8.10 but had to downgrade due to some major AirGroup issues. So the issue occurred on AOS 8.7 and 8.10, and on both our old HP 8212zl core as well as our new Aruba 6405 core. The NAC and all wifi MM and MDs are connected directly to the core via 10GB fiber.

Could there be packet loss inbetween the VLANs? Could the controllers be dropping packets intermittently? 

Any ideas? I'm desperate at this point.

I wouldn't have anything on VLAN 1.  If I wasn't running MSTP, I wouldn't have VLAN 1 enabled on any port.

Network devices, servers, etc., all get assigned to a VLAN other than VLAN 1.

Thanks. This is helpful. We have our core and all of our edge switches on VLAN 1, along with our NAC. So is it your recommendation is to only have switches on VLAN 1 and keep other servers off of it?

Original Message:
Sent: Dec 02, 2024 10:25 AM
From: chulcher
Subject: Possible inter-VLAN packet loss

Can't say that I have a specific recommendation to make on this, but every time I see someone using VLAN 1 in production for anything there seems to be problems associated with doing so.  Too many devices treat VLAN 1 differently from other VLANs for me to ever trust using that VLAN for anything other than the native VLAN when running MSTP.

------------------------------
Carson Hulcher, ACEX#110

Original Message:
Sent: Nov 29, 2024 10:17 AM
From: nkuhl30
Subject: Possible inter-VLAN packet loss

Hello everyone,

I posted about this over the Summer of 2024 but never got anywhere with TAC, and the symptoms stopped once the students arrived back to campus, so I thought I'd reach out again for a fresh look. With the assistance of our Aruba SE, we upgraded our core switch from an HP 8212zl to an Aruba 6405 back in June 2024. After about a day, we noticed that our NAC (FortiNAC) was intermittently failing SNMP polls and pings to our two Aruba 7210 wireless controllers. If we check the controllers, they never went down and neither did any of the APs. The same goes for the core switch. There's no evidence of packet loss on the ports and the ports never lost connection to the core.

Our core is on VLAN 1 and our Aruba controllers are on VLAN 40 with no ACLs or firewall in between. After spending a lot of time looking at logs on the wireless side, I had noticed that our mobility manager, that manages the two controller cluster, would drop connection to each controller every 51 minutes, at different times in the hour. After 10 seconds, the connection would re-establish. Our mobility master was also on VLAN 1, not on VLAN 40 with the controllers.

Thinking that the new core is doing something different than the old HP, we went down that rabbit hole but didn't come up with anything significant. However, I did look back at a TAC case from 2023 where we pulled tech support logs, and sure enough, the MM was dropping connection to both controllers at the same rate of every 51 minutes and we didn't know it. And this was happening on the old core switch so that rules out the 6405 causing the issue.

With the help of our Aruba SE, we moved the MM to VLAN 40 and the 51 min connection drop between the MM and the controllers has stopped. Now FortiNAC had continued to alert us to SNMP poll failures (contact lost) up until students started arriving back on campus in late August. At that point, the FortiNAC alerts stopped and everything was normal again. Well that changed last week when all of the students left campus for Thanksgiving break and the issue has returned.

What could possibly be the issue here? We are running an old version of Aruba OS (8.7.1.11) on the MM and MDs and I thought that an upgrade may help. However, the tech support logs taken back in the Summer of 2023 were taken when we dipped out toes into AOS 8.10 but had to downgrade due to some major AirGroup issues. So the issue occurred on AOS 8.7 and 8.10, and on both our old HP 8212zl core as well as our new Aruba 6405 core. The NAC and all wifi MM and MDs are connected directly to the core via 10GB fiber.

Could there be packet loss inbetween the VLANs? Could the controllers be dropping packets intermittently? 

Any ideas? I'm desperate at this point.