Wireless Access

Reply
Highlighted
Regular Contributor I

Why new controllers stopped communicating with Mobility Masters

We have 2 x Mobility Masters and 2 Local Controlelrs that are setup in our Main Headquarters location.  Lets call the Headquarters 'Location - A'.

 

We have 2 x Local Controllers at another location called 'Location- B'.  The local controlelrs from Location - A & B are communicating just fine with the Mobility Masters and MM Virtual I.P. Address in Location - A.

 

We just purchased 2 x Aruba 7010 controllers for another location called 'Location - C'.  I setup the Location - C controllerds and they initially connected to the Mobility Masters.  100% verified and update successfull.  PSK with MAC Address while using the MM Virtual IP Address.

 

Then I left for the evening and when I returned the next morning we had a firewall failure at Location - C's and all communication with Location - A was down.  Location - C connects to Location - A via Dedicated MPLS circuit.  After bringing the Firewall back up (Power off and on) the communication was restored with the exception of the local controllers not being able to communicate via port 4500 to the mobility masters in Location - A.  

 

I spoke with Aruba support and they informed me that the PSK configurations on the Mobility Masters and controllers appear to be correct.  The problem is that the Mobility Masters are not receving any port 4500 communications from the Location - C controllers.  

 

The Firewalls on both ends state that communocation on port 4500 is being sent from Location - C and is indeed received on Location - A; but, Location - A is not sending back communication to the source.

 

Statement1:  What is the problem:

Answer1: The Location - C controllers are not able to communicate or synch up with the Mobility Masters located in Location - A.  

 

Statement2: Why is that problem happening?

Answer2:  The Mobility Masters are not receiving any communications on port 4500 from the Location - C Aruba local controllers.  Support stated that is required to communiocate with the Mobility Masters.

 

Question1:  How can we fix the communication problems and sych up the Aruba Local Controllers in Location - C with the Mobility Masters in Locaiton - A.

 

 

Regular Contributor I

Re: Why new controllers stopped communicating with Mobility Masters

I referenced some of the trouble-shooting commands from:  https://community.arubanetworks.com/t5/Controller-Based-WLANs/Understanding-and-Troubleshooting-IPSec-issues/ta-p/240527

 

1.  Basically the mobility master does not see any port 4500 traffic coming from the Location - C controllers.  Even though our FIrewall Admins are stating that port 4500 traffic is passing through the firewall on Locaiton - A.

 

2.  If I execute: MM) [mynode] #show datapath session table XXX.Virtual_IP_Address | include 4500

The IP addresses fro the local controllers at 'Location - C' are not listed for the source or destination.  Other locations are working just fine.

 

3.  What is interesting is when I execute >show datapath session table MM_V.I.P. | include 4500     from the controllers in Location - C I see packets going to the MM_V.I.P. but the Destination field is shown as 'local'.

 

When I execute the same command from the other local controllers (at other sites) the Destination field is shown as '0/0/0'.  I see traffic from port 4500 being sent to the MM_V.I.P.  just not returned from the MM_V.I.P.  Other commands are showing that the 4500 traffic is not being receved by the MM even though the Location - A firewall is shwoing the 4500 traffic from Location - C coming thhrough.

 

I have a couple of theries to what is going on; but, I would love to hear back from the Aruba Experts.  Keep in mind that these are brand new controllers.  I am not sure if the licesning was added fro the controlelrs yet... perhaps that is preventing some functionality.  Perhaps licesing for the new 7010 needs to be configured?  How can I verify that?

Super Contributor I

Re: Why new controllers stopped communicating with Mobility Masters

At the MM's you are using a VIP. Are you sure that the VRRP instance ID is unique and not used by other systems in the network?

Willem Bargeman ACMX#935 | ACCX #822

Please give me kudos if my post was useful!
If your issue is solved mark the post as solution!
Regular Contributor I

Re: Why new controllers stopped communicating with Mobility Masters

MM VRRP instance.  Do you mean in teh web User Interface: MM - Configuration - Services - Redundancy - Virtual Router Section?

 

Virtual Router Name = 1

I did not set that up; but, that name '1' is not used for anoy other VRRP at teh Local controlelr level.  

 

Where should I find the information you are requesting?  I did not setup the Mobility Masters; but, the above location is where I see the MM Virtual IP Address.  That is where the local controllers are tryiing to connect to.

 

At the local controller level, the router names are using different ID numbers at every location. 4, 5, etc.  Like I mentioned before the connection was good for a littel while and hten just stopped.

Super Contributor I

Re: Why new controllers stopped communicating with Mobility Masters

Well, if another device in the same L2 domain is using the same vrrp id there is a duplicate MAC address. This can result in traffic issues. Indeed it was working before but still this can be a issue.

If you not see the traffic reaching the MM, also not in the session table, there is probably any other issue outside the MM and local controller.

Willem Bargeman ACMX#935 | ACCX #822

Please give me kudos if my post was useful!
If your issue is solved mark the post as solution!
Regular Contributor I

Re: Why new controllers stopped communicating with Mobility Masters

To be clear, each site/location has their own Layer-2 Wireless Vlan dedicated to that site. The Wireless VLan ID is the same number; but, the IP Scheme is different at every site.  The Wireless Vlan is where the Aruba Local controllers and Aruba APs reside.  Hence the local controller may be on VLan-101 at every site; but, the IP scheme's are different and the VRRP ID #'s are different at every site.

 

The Mobility Master Virtual Servers reside on the Infrastructure VLan which is separate from Location - A's Wireless VLan.  

 

Like I said, Location B.'s local controllers can communicate with Location - A's Mobility Masters just fine.

 

"If you not see the traffic reaching the MM, also not in the session table, there is probably any other issue outside the MM and local controller."   Any ideas regarding the 4500 traffic from Location - C and MM?  Is it a configuration that must be done on the MM to allow Location - Local Controller's access to the MM?

 

Firewall communication? Licensing problems pn the MM that block functionality? 

 

Equally important, I did try a 'write erase' on 1 of the lcoal controllers at the Locaiton - C site.  That did not work to restore port 4500 traffic communication with the MM from Location - A?

Frequent Contributor II

Re: Why new controllers stopped communicating with Mobility Masters

As the timing of the issue relates to the firewall reboot, it seems possible that the firewall had been explicitly configured to allow MC->MM communications but that particular policy was lost at reboot time (e.g. firewall config was not saved to nvram)

Regular Contributor I

Re: Why new controllers stopped communicating with Mobility Masters

I still do not know why the MM's are treating the controllers at Location - C differently.

 

1.  Is it a firewall problem, even though the firewall at 'Locaiton A' sees communication passign through from the local  controllers at Locaiton C trying to go to the MM Virtual V.I.P.?  

     a.  Do we need to enter license keys for the new 7010 controller to communicate with the Mobility Master?  

 

2.  Maybe we can trace the 4500 traffic from the local controllers after they arrive at the Locaiton A office.

 

3.  Does the Mobility Master disable a local controller if they are disconnected after a specific periof of time?

     a.  Keep in mind Locaiton C was disconnected for hours and then reconnected after teh Firewall reboot.

     b.  I have seen similar security measures happen with other applications that require devices to be manually re-enabled.

 

4,.  Why is it when I execute >show datapath session table MM_V.I.P. | include 4500<enter>     from the controllers in Location - C I see packets going to the MM_V.I.P. but the Destination field is shown as 'local'.

 

When I execute the same command from the other local controllers (at other sites) the Destination field is shown as '0/0/0'.  Why is the festination different and showing up as 'local' instead of 0/0/0?

 

Source IP | Destination IP | Prot | SPort | DPort | Cntr | Prio | ToS | Age | Destination | TAge | Packets | Bytes | Flags | CPU ID

 

5.  Can there be something at the Mobility Master end that is required to allow 4500 traffic to communicate again?  Ironically just a few days ago (after the 4500 traffic problem) the MMs were restarted by mistake and the backup MM is now the main MM.  A restart of the MM's did not cure the problem.  I am wondering if I should restart teh backup MM so I can get the main MM back as primary?

 

6.  Might this have anyhting to do with connecting Access Point Ap 345 to the Wireless Controller VLan?  These 345 booted up and automatically updated themselves to the same versrion as the controller (8.3.0.3).

    a.  I was abel to see the bootup and update process while I was connectged to the AP's thorugh a lcoal SSH connection.  I beleive the AP's came with verison 8.0 and then recognized the controller had version 8.3 and initiated an update automatically.

     b.  At the other sites we used IAP - 325 that came with version 6.4 and I needed to manually update to version 6.5 and then manually convert the IAP to campus mode.

     c.  This is prbably a non-issue; but, I did not bring it up in the past.  

     d.  Should we un-plug the AP-345 to see if that helps?

      e. Model: AP-345, SNs: CNFZK5110Y, CNFZK5114C, CNFZK51134, CNFZK5112M, more of the same.

 

7. Is tehre any licensing that needs to be entered on the MM end to work correctly with the new Local Controllers at Location C?

 

8.  Quite simply the communcation with the MMs from Locaiton C was working before the Firewall problem and now it does not.  I even did a 'write erase' on 1 of the controllers and that controller now does not even see the inherited configuraiton that the other Location C controller already downloaded.  Because communication with the MMs is down.  This proves teh problem is not with teh configuration on the local controller.

     a.  Why would the MM not receive port 4500 communication from the local controllers at a diffeent site?

     b.  Is there any configuration that is required on the MM besides the MAC address and its PSK (verified with >show local-peer-mac<enter>

     c.  Does the MM require licesing information to work correctly with the new Local controlers and its Access Points?

Regular Contributor I

Re: Why new controllers stopped communicating with Mobility Masters

It was deterined that the packets are being send from the

 

1.  MD in the Locaiton C

2.  LAN Switch in Locaiton C

3.  FW in Locaiton C

4.  Dedicated MPLS to Locaiton A

5.  FW in Location A

6.  LAN Switch in Locaiton A

..... never gets delivered to MM in Locaiton A.

 

Hence, it appears that there is a problem with the IPSec tunnel on the MM's.  That is why the port 4500 Packets are not returning from Locaiton A.  It was said that the MM must have some problems with the Locaiton C MDs, after the communicaiton downage last week.  How can we fix this?

 

We have tried to change the settings on the MD side.  We have doens a write erase and have changed the PSKwithMAC settings to PSKwithIP settings on the MD side.  Then cahnged it accordingl on the MM Web user interface.  The key piece of information is that the packets are passing through the Location A firewall and then to the Main LAN Switch.... it is not being receved my the MM.  What can be done on the MM side to clear up any settings for teh IP Sec communications?

 

I am afraid that I wll need to do another write erase on both controllers this time and use a different IP address for each controller?

 

Any ideas?  Theoretically speaking this might happen at any locatin if there is a bad connection.  We will not be able to change teh IP addresses so easily if the Aruba Wireless is fully in production.


Regular Contributor I

Re: Why new controllers stopped communicating with Mobility Masters

The reason the MDs stopped talking to the MMs was becuase the MD's uplink port (0/0/0) was set in trunk mode.  The cross connecting switch port was an access port.

 

That is why the show datapath sessio table output showed 'local' instead of port 0/0/0 for its 'Destination' field.

 

After changing the LAN switch port setting to trunk the communicatio to the MM was restored.  

Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: