Wired Intelligent Edge

Hello,

I'm new to Aruba switches and the Aruba community and looking for a few pointers to deal with two or three problems. Sorry this is so long. I figured I should avoid initial questions and give as much detail as I could.

Basic setup: We are an apartment/condo with 56 apartment units all wired to a pair of 48 port/4 sfp+ port VSF stacked Aruba 2930F switches. There is either a single computer, an unmanged 5 port switch, or a home router in each of the 56 units and a few other spots around the building. No other managed switches at all, so the 2930F stack is acting basically as our core and edge of our network.

The 2930F switches have (will have) a pair of 10 GB cables trunked between them for the VSF link and another pair of 10 GB cables trunked between the first 2930F and the firewall/router, all mainly for redundancy not bandwidth. (Right now LACP LAGGs trunk pairs are setup between the two switches and from the switch stack to the firewall/router, but only one cable is in place for each trunk...more cables on order.)

We are using a single IP space 10.1.0.0/16, with a /24 in there nominally allocated to each of the 56 units to do with as they please.

 

The plan is to use 8 tagged Private VLANs with a tagged community PVLAN for each of the 56 units (and one more for a shared computer nook area) and including the uplink ports (trk1) on the 8 primary VLANs being promiscuous ports. (I'm using 8 Private VLANS as each only allows 8 Community VLANs under it.)

Eventually, we want to use DHCP to assign a unique part of our /16 network to each unit so we are using a community PVLAN for each unit given the restriction on only having a single Isolated PVLAN per primary private VLAN.) As I read it, I can assign a specific DHCP range to each Community VLAN but can't do that directly to a single port.

At this point, I'm testing this all out on PVLAN 100 with community PVLAN 99 under it. (I've got other PVLANs 10-16 setup with 56 Community VLANs under them, but havent added the trk1 to those yet so not testing those yet.)

At this point, I've added the trk1 to PVLAN 100 and removed one of the ports (1/46) that is part of Community PVLAN 99 from the DEFAULT_VLAN 1 in order to isolate it from the other ports. (Before I remove 1/46 from VLAN 1 I can ping from it to other ports not in VLAN 99 so I'm not isolated.)

(1) My confusion is that when I remove a port from DEFAULT_VLAN 1  then I can (as expected) no longer ping other ports, but I also can't reach the gateway/internet from that port.

What am I missing here?

Thoughts are:
+ Everything is in one IP space, even with the 64 VLANs, so IP routing on the switch should not be needed, right?
+ When all ports are still part of the DEFAULT_VLAN 1, all units can get to the router/firewall/gateway/internet 10.1.10.1 just fine with no special VLAN setup on the router. Howeve, maybe my point I've missed is that when I move ports out of DEFAULT_VLAN 1 and into ONLY tagged VLANs 99, 101, 102, etc. I need to create a matching new port on the router for that VLAN with associated firewall rules?
+ If not, what am I missing in my Private VLAN with Community VLANs and uplink promiscuous ports setup?

(2) Since we have only 1 switch effectivly with our two 2930F switches stacked, we are not using STP, though I had initially enabled it when we thought we'd have another managed switch in the mix. We do have unmanaged switches hanging off ports at times, and untrusted users, so I enabled loop protection in an attempt to avoid our network going down if someone plugs in a loop on a unmanaged switch that hangs off one of the 2930F ports.

I configured the 2930F stack with:
loop-protect 1/1-1/46,2/1-2/46
loop-protect trap loop-detected
loop-protect mode vlan
loop-protect vlan 10-16,100
loop-protect disable-timer 300

(If you look at the current full config the disable timer is now 3600 to help deal with what I'm about to ask about...also, when this test was done all ports were part of the port-based UNtagged DEFAULT_VLAN 1.)

I plugged in a Ethernet cable into a unmanaged switch that hangs off port 1/46, a loop was created as expected, network response dropped like a rock everywhere on the network, a loop was detected by loop protect, port 1/46 was shut down, and network responsiveness returned to normal.

Show loop on the 2930F showed the 1/46 port as down (the network responded fine again as soon as the port was down) and I watched/repeated the show loop command for 5 minutes at which point, as expected, the port came back up. However, the port then stayed up and the network response went to near nothing.

Why didn't, as implied in ArubaOS Advanced Traffic Management Guide and as I see people discuss here, the 2930F detect the still existing loop 5 seconds later and bring down port 1/46 again?

As a workaround, I've set the disable timer to 3600 so if someone makes a loop it will be down for an hour and hopefully they will contact me. (Worst case when it comes up again and the network dies I can check the switch logs to find them.) Another option I'm seeing is to set the disable timer to the default of 0 so once the port goes down it stays down until I manually bring it up after pulling the loop.

(3) Additionally, the ArubaOS Advanced Traffic Management Guide says, "The port will not transmit loop protection packets unless it is a member of an untagged VLAN. If a port is only a member of tagged VLANs, the loop protection packets are not transmitted." Each of my ports exposed to end users is in a tagged VLAN, but I wasn't planning on having them in any UNtagged VLAN. (In fact, question 1 above was specifically how I can remove them from DEFAULT_VLAN 1 and still get to the internet.) I don't want everything on a single VLAN becuase I want each of the 56 units' port to NOT be able to reach any other ports except the uplink ports. How can I meet the requirement to have each loop protected port on a UNtagged VLAN without creating another extra 56 single port UNtagged VLANs for no reson other that to make loop protect work?

This is my current configuration with the runs of 56 of the same VLAN setup, VLAN naming, ACL setup, ACL assignment, etc. clipped and replaced with elipsis. The full config without modification is attached.

Thanks in advance for any help or pointers for the new guy...

 

Trimmed Config: 

; hpStack_WC Configuration Editor; Created on release #WC.16.06.0006
; Ver #13:4f.f8.1c.9b.3f.bf.bb.ef.7c.59.fc.6b.fb.9f.fc.ff.ff.37.ef:05
hostname "EVC HPE Aruba switch 1"
vsf
   enable domain 1
   member 1
      type "JL256A" mac-address f40343-10a3c0
      priority 245
      link 1 1/51-1/52
      link 1 name "I-Link1_1"
      link 2 name "I-Link1_2"
      exit
   member 2
      type "JL256A" mac-address ecebb8-35a000
      priority 10
      link 1 2/51-2/52
      link 1 name "I-Link2_1"
      link 2 name "I-Link2_2"
      exit
   port-speed 10g
   exit
console idle-timeout 3600
fault-finder broadcast-storm sensitivity high
fault-finder bad-driver sensitivity high
fault-finder bad-transceiver sensitivity high
fault-finder bad-cable sensitivity high
fault-finder too-long-cable sensitivity high
fault-finder over-bandwidth sensitivity high
fault-finder loss-of-link sensitivity high
fault-finder duplex-mismatch-hdx sensitivity high
fault-finder duplex-mismatch-fdx sensitivity high
fault-finder link-flap sensitivity high
trunk 1/49-1/50 trk1 lacp
trunk 2/49-2/50 trk2 lacp
password minimum-length 8
timesync ntp
ntp unicast
ntp server 10.1.10.1 iburst
ntp enable
time daylight-time-rule continental-us-and-canada
time timezone -300
web-management ssl
web-management idle-timeout 3600
ip access-list extended "system_ports_in"
     10 deny udp 0.0.0.0 255.255.255.255 eq 67 0.0.0.0 255.255.255.255 log
     20 permit ip 10.1.115.0 0.0.0.255 0.0.0.0 255.255.255.255
     30 permit ip 10.1.0.0 0.0.0.255 0.0.0.0 255.255.255.255
     40 permit ip 10.1.1.0 0.0.0.255 0.0.0.0 255.255.255.255
     50 permit ip 10.1.2.0 0.0.0.255 0.0.0.0 255.255.255.255
     60 permit ip 10.1.3.0 0.0.0.255 0.0.0.0 255.255.255.255
     70 permit ip 10.1.4.0 0.0.0.255 0.0.0.0 255.255.255.255
     80 permit ip 10.1.5.0 0.0.0.255 0.0.0.0 255.255.255.255
     90 permit ip 10.1.6.0 0.0.0.255 0.0.0.0 255.255.255.255
     100 permit ip 10.1.7.0 0.0.0.255 0.0.0.0 255.255.255.255
     110 permit ip 10.1.8.0 0.0.0.255 0.0.0.0 255.255.255.255
     120 permit ip 10.1.9.0 0.0.0.255 0.0.0.0 255.255.255.255
     130 permit ip 10.1.10.0 0.0.0.255 0.0.0.0 255.255.255.255
     140 permit ip 10.1.15.0 0.0.0.255 0.0.0.0 255.255.255.255
     150 deny ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255 log
   exit
ip access-list extended "unit_101_in"
     10 deny udp 0.0.0.0 255.255.255.255 eq 67 0.0.0.0 255.255.255.255 log
     20 permit ip 10.1.101.0 0.0.0.255 0.0.0.0 255.255.255.255
     30 deny ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255 log
   exit
ip access-list extended "unit_102_in"
     10 deny udp 0.0.0.0 255.255.255.255 eq 67 0.0.0.0 255.255.255.255 log
     20 permit ip 10.1.102.0 0.0.0.255 0.0.0.0 255.255.255.255
     30 deny ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255 log
   exit
ip access-list extended "unit_103_in"
     10 deny udp 0.0.0.0 255.255.255.255 eq 67 0.0.0.0 255.255.255.255 log
     20 permit ip 10.1.103.0 0.0.0.255 0.0.0.0 255.255.255.255
     30 deny ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255 log
   exit
.
.
.
ip access-list extended "unit_416_in"
     10 deny udp 0.0.0.0 255.255.255.255 eq 67 0.0.0.0 255.255.255.255 log
     20 permit ip 10.1.156.0 0.0.0.255 0.0.0.0 255.255.255.255
     30 deny ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255 log
   exit
ip access-list extended "unit_419_in"
     10 deny udp 0.0.0.0 255.255.255.255 eq 67 0.0.0.0 255.255.255.255 log
     20 permit ip 10.1.159.0 0.0.0.255 0.0.0.0 255.255.255.255
     30 deny ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255 log
   exit
ip default-gateway 10.1.10.1
interface 1/1
   ip access-group "unit_201_in" in
   name "201"
   exit
interface 1/2
   ip access-group "unit_101_in" in
   name "101"
   exit
interface 1/3
   ip access-group "unit_202_in" in
   name "202"
   exit
interface 1/4
   ip access-group "unit_102_in" in
   name "102"
   exit
interface 1/5
   ip access-group "unit_204_in" in
   name "204"
   exit
interface 1/6
   ip access-group "unit_103_in" in
   name "103"
   exit
.
.
.
interface 1/31
   ip access-group "system_ports_in" in
   exit
interface 1/32
   ip access-group "system_ports_in" in
   exit
interface 1/33
.
.
.
interface 1/45
   ip access-group "system_ports_in" in
   exit
interface 1/46
   ip access-group "system_ports_in" in
   name "Computer Nook"
   exit
interface 1/49
   name "pfSense3 Trk1"
   exit
interface 1/50
   name "pfSense3 Trk1"
   exit
interface 1/51
   name "VSF_link_1"
   exit
interface 1/52
   name "VSF_link_1"
   exit
.
.
.
interface 2/49
   name "pfSense3 backup Trk2"
   exit
interface 2/50
   name "pfSense3 backup Trk2"
   exit
interface 2/51
   name "VSF_link_1"
   exit
interface 2/52
   name "VSF_link_1"
   exit
snmp-server community "public" operator
snmp-server contact “********” location
 "EVC Main Electrical room, 1st floor "
vlan 1
   name "DEFAULT_VLAN"
   untagged 1/1-1/48,2/1-2/48,Trk1-Trk2
   ip address 10.1.10.4 255.255.0.0
   exit
vlan 10
   name "Units_101-108_PVLAN"
   private-vlan primary
   private-vlan community 101-108
   no ip address
   exit
vlan 11
   name "Units_113-201_PVLAN"
   private-vlan primary
   private-vlan community 113-119,201
   no ip address
   exit
vlan 12
   name "Units_202-210_PVLAN"
   private-vlan primary
   private-vlan community 202,204-210
   no ip address
   exit
vlan 13
   name "Units_211-304_PVLAN"
   private-vlan primary
   private-vlan community 121-124,211-212,216,219
   no ip address
   exit
vlan 14
   name "Units_305-312_PVLAN"
   private-vlan primary
   private-vlan community 125-132
   no ip address
   exit
vlan 15
   name "Units_313-401_PVLAN"
   private-vlan primary
   private-vlan community 133-139,141
   no ip address
   exit
vlan 16
   name "Units_407-419_PVLAN"
   private-vlan primary
   private-vlan community 147-150,152,154,156,159
   no ip address
   exit
vlan 99
   name "EVC_System_PVLAN_community"
   tagged
 1/22,1/27,1/29,1/31-1/46,2/19,2/21,2/23,2/25,2/27,2/29,2/31,2/33,2/35,2/37,2/39
-2/46
   no ip address
   exit
vlan 100
   name "EVC_System_PVLAN"
   private-vlan primary
   private-vlan community 99
   no ip address
   exit
vlan 101
   name "Unit_101_PVLAN_community"
   tagged 1/2
   no ip address
   exit
vlan 102
   name "Unit_102_PVLAN_community"
   tagged 1/4
   no ip address
   exit
.
.
.
vlan 216
   name "Unit_216_PVLAN_community"
   tagged 1/23
   no ip address
   exit
vlan 219
   name "Unit_219_PVLAN_community"
   tagged 1/25
   no ip address
   exit
spanning-tree Trk1 priority 4
spanning-tree Trk2 priority 4
no tftp server
loop-protect 1/1-1/46,2/1-2/46
loop-protect trap loop-detected
loop-protect mode vlan
loop-protect vlan 10-16,100
loop-protect disable-timer 3600
no autorun
no dhcp config-file-update
no dhcp image-file-update
trunk-load-balance L4-based
password manager
password operator

 

That is a lot of information.

 

On your loop protect not kicking in again after the recovery timer, please open a TAC case as that doesn't sound right to me.

 

Given all the access-lists that you created to control traffic, you might consider moving to a L3 design rather than using private ports. You can still decide to do routing on the switch or leave the switch L2 and create a VLAN for each unit on your pfSense to make sure they don't intercommunicate. Then on your uplinks to the firewall, you have all the VLANs tagged, and they will be untagged on just one interface of your switches. I would go for the latter option personally, with inter VLAN routing on the firewall; and you can run DNS and DHCP there as well.

 

With that design, even if there is a loop in the network, it doesn't hurt as you don't L2 loop but just connect two L3 subnets which go to the firewall and doesn't have any performance impact on the switch (nor on the firewall).

Thanks for the tips. I’ll open the case for the loop protect behavior.

The L2 only suggestion is a possibility. One downside I see is making and handling 57 interfaces in pfSense for the 57 VLANs.

I’d still like to understand why the Private PVLANs aren’t working/able to reach the gateway as they should when I remove their port from the DEFAULT_VLAN 1, if anyone knows what my mistake is in this current architecture.

One point for keeping L3 on the switch is that I hope to eventually lock down the IP addresses with Dynamic IP Protection and Dynamic ARP Protection to avoid someone setting their unit’s router’s IP address to the gateway address, which has happened in the past. Users! (Basically I need to eventually end up operating as if I’m in a hostil environment.)

Adding an update on the loop protection issue.

 

I've done several additional rounds of testing and found that loop protection does work...eventually.

 

Creating a loop in an unmanaged switch off of port 1/46 loop brings down port 1/46 and, after the set number of disable seconds, brings the port back up. Loop protection DOES then bring the port down again, but only after about 10 minutes of active looping and impacted network operations.

 

After loop protect detects the loop a second time and brings the port down a second time (and then after the set number of disable seconds) it brings the port back up again and then again after about another 10 minutes brings the port down again. So the loop-protect is working, but it is taking about 10 minutes to disable the port after the first immediate disabling and eventual enabling again.

 

Now on to trying to get Private VLANs working as described in the Advance Traffic Management Guide.

Final update before I mark this as (sort of) resolved.

 

First, my config had all ports in VLANs as tagged. The single port in each VLAN to which a device or individual apartment's home router is connected should be UNtagged since those devices don't understand tagged VLANs.

 

I still can't get Private VLANs with Community VLANs under them to work so I've abandoned those and gone to one standard VLAN per apartment with each trunked to the firewall and attached to their own virtual interface on the firewall.

 

Working OK so far for internet access, though I can't get devices on the VLANs to successfully get a DHCP address from the DHCP server running on the pfSense firewall. One step at a time...

 

Hope this helps someone someday.