Wireless Access

We have 2 x Mobility Masters and 2 Local Controlelrs that are setup in our Main Headquarters location.  Lets call the Headquarters 'Location - A'.

 

We have 2 x Local Controllers at another location called 'Location- B'.  The local controlelrs from Location - A & B are communicating just fine with the Mobility Masters and MM Virtual I.P. Address in Location - A.

 

We just purchased 2 x Aruba 7010 controllers for another location called 'Location - C'.  I setup the Location - C controllerds and they initially connected to the Mobility Masters.  100% verified and update successfull.  PSK with MAC Address while using the MM Virtual IP Address.

 

Then I left for the evening and when I returned the next morning we had a firewall failure at Location - C's and all communication with Location - A was down.  Location - C connects to Location - A via Dedicated MPLS circuit.  After bringing the Firewall back up (Power off and on) the communication was restored with the exception of the local controllers not being able to communicate via port 4500 to the mobility masters in Location - A.  

 

I spoke with Aruba support and they informed me that the PSK configurations on the Mobility Masters and controllers appear to be correct.  The problem is that the Mobility Masters are not receving any port 4500 communications from the Location - C controllers.  

 

The Firewalls on both ends state that communocation on port 4500 is being sent from Location - C and is indeed received on Location - A; but, Location - A is not sending back communication to the source.

 

Statement1:  What is the problem:

Answer1: The Location - C controllers are not able to communicate or synch up with the Mobility Masters located in Location - A.  

 

Statement2: Why is that problem happening?

Answer2:  The Mobility Masters are not receiving any communications on port 4500 from the Location - C Aruba local controllers.  Support stated that is required to communiocate with the Mobility Masters.

 

Question1:  How can we fix the communication problems and sych up the Aruba Local Controllers in Location - C with the Mobility Masters in Locaiton - A.

 

 

I referenced some of the trouble-shooting commands from:  https://community.arubanetworks.com/t5/Controller-Based-WLANs/Understanding-and-Troubleshooting-IPSec-issues/ta-p/240527

 

1.  Basically the mobility master does not see any port 4500 traffic coming from the Location - C controllers.  Even though our FIrewall Admins are stating that port 4500 traffic is passing through the firewall on Locaiton - A.

 

2.  If I execute: MM) [mynode] #show datapath session table XXX.Virtual_IP_Address | include 4500

The IP addresses fro the local controllers at 'Location - C' are not listed for the source or destination.  Other locations are working just fine.

 

3.  What is interesting is when I execute >show datapath session table MM_V.I.P. | include 4500     from the controllers in Location - C I see packets going to the MM_V.I.P. but the Destination field is shown as 'local'.

 

When I execute the same command from the other local controllers (at other sites) the Destination field is shown as '0/0/0'.  I see traffic from port 4500 being sent to the MM_V.I.P.  just not returned from the MM_V.I.P.  Other commands are showing that the 4500 traffic is not being receved by the MM even though the Location - A firewall is shwoing the 4500 traffic from Location - C coming thhrough.

 

I have a couple of theries to what is going on; but, I would love to hear back from the Aruba Experts.  Keep in mind that these are brand new controllers.  I am not sure if the licesning was added fro the controlelrs yet... perhaps that is preventing some functionality.  Perhaps licesing for the new 7010 needs to be configured?  How can I verify that?

At the MM's you are using a VIP. Are you sure that the VRRP instance ID is unique and not used by other systems in the network?

MM VRRP instance.  Do you mean in teh web User Interface: MM - Configuration - Services - Redundancy - Virtual Router Section?

 

Virtual Router Name = 1

I did not set that up; but, that name '1' is not used for anoy other VRRP at teh Local controlelr level.  

 

Where should I find the information you are requesting?  I did not setup the Mobility Masters; but, the above location is where I see the MM Virtual IP Address.  That is where the local controllers are tryiing to connect to.

 

At the local controller level, the router names are using different ID numbers at every location. 4, 5, etc.  Like I mentioned before the connection was good for a littel while and hten just stopped.

Well, if another device in the same L2 domain is using the same vrrp id there is a duplicate MAC address. This can result in traffic issues. Indeed it was working before but still this can be a issue.

If you not see the traffic reaching the MM, also not in the session table, there is probably any other issue outside the MM and local controller.

To be clear, each site/location has their own Layer-2 Wireless Vlan dedicated to that site. The Wireless VLan ID is the same number; but, the IP Scheme is different at every site.  The Wireless Vlan is where the Aruba Local controllers and Aruba APs reside.  Hence the local controller may be on VLan-101 at every site; but, the IP scheme's are different and the VRRP ID #'s are different at every site.

 

The Mobility Master Virtual Servers reside on the Infrastructure VLan which is separate from Location - A's Wireless VLan.  

 

Like I said, Location B.'s local controllers can communicate with Location - A's Mobility Masters just fine.

 

"If you not see the traffic reaching the MM, also not in the session table, there is probably any other issue outside the MM and local controller."   Any ideas regarding the 4500 traffic from Location - C and MM?  Is it a configuration that must be done on the MM to allow Location - Local Controller's access to the MM?

 

Firewall communication? Licensing problems pn the MM that block functionality? 

 

Equally important, I did try a 'write erase' on 1 of the lcoal controllers at the Locaiton - C site.  That did not work to restore port 4500 traffic communication with the MM from Location - A?

As the timing of the issue relates to the firewall reboot, it seems possible that the firewall had been explicitly configured to allow MC->MM communications but that particular policy was lost at reboot time (e.g. firewall config was not saved to nvram)

I still do not know why the MM's are treating the controllers at Location - C differently.

 

1.  Is it a firewall problem, even though the firewall at 'Locaiton A' sees communication passign through from the local  controllers at Locaiton C trying to go to the MM Virtual V.I.P.?  

     a.  Do we need to enter license keys for the new 7010 controller to communicate with the Mobility Master?  

 

2.  Maybe we can trace the 4500 traffic from the local controllers after they arrive at the Locaiton A office.

 

3.  Does the Mobility Master disable a local controller if they are disconnected after a specific periof of time?

     a.  Keep in mind Locaiton C was disconnected for hours and then reconnected after teh Firewall reboot.

     b.  I have seen similar security measures happen with other applications that require devices to be manually re-enabled.

 

4,.  Why is it when I execute >show datapath session table MM_V.I.P. | include 4500<enter>     from the controllers in Location - C I see packets going to the MM_V.I.P. but the Destination field is shown as 'local'.

 

When I execute the same command from the other local controllers (at other sites) the Destination field is shown as '0/0/0'.  Why is the festination different and showing up as 'local' instead of 0/0/0?

 

Source IP | Destination IP | Prot | SPort | DPort | Cntr | Prio | ToS | Age | Destination | TAge | Packets | Bytes | Flags | CPU ID

 

5.  Can there be something at the Mobility Master end that is required to allow 4500 traffic to communicate again?  Ironically just a few days ago (after the 4500 traffic problem) the MMs were restarted by mistake and the backup MM is now the main MM.  A restart of the MM's did not cure the problem.  I am wondering if I should restart teh backup MM so I can get the main MM back as primary?

 

6.  Might this have anyhting to do with connecting Access Point Ap 345 to the Wireless Controller VLan?  These 345 booted up and automatically updated themselves to the same versrion as the controller (8.3.0.3).

    a.  I was abel to see the bootup and update process while I was connectged to the AP's thorugh a lcoal SSH connection.  I beleive the AP's came with verison 8.0 and then recognized the controller had version 8.3 and initiated an update automatically.

     b.  At the other sites we used IAP - 325 that came with version 6.4 and I needed to manually update to version 6.5 and then manually convert the IAP to campus mode.

     c.  This is prbably a non-issue; but, I did not bring it up in the past.  

     d.  Should we un-plug the AP-345 to see if that helps?

      e. Model: AP-345, SNs: CNFZK5110Y, CNFZK5114C, CNFZK51134, CNFZK5112M, more of the same.

 

7. Is tehre any licensing that needs to be entered on the MM end to work correctly with the new Local Controllers at Location C?

 

8.  Quite simply the communcation with the MMs from Locaiton C was working before the Firewall problem and now it does not.  I even did a 'write erase' on 1 of the controllers and that controller now does not even see the inherited configuraiton that the other Location C controller already downloaded.  Because communication with the MMs is down.  This proves teh problem is not with teh configuration on the local controller.

     a.  Why would the MM not receive port 4500 communication from the local controllers at a diffeent site?

     b.  Is there any configuration that is required on the MM besides the MAC address and its PSK (verified with >show local-peer-mac<enter>

     c.  Does the MM require licesing information to work correctly with the new Local controlers and its Access Points?

It was deterined that the packets are being send from the

 

1.  MD in the Locaiton C

2.  LAN Switch in Locaiton C

3.  FW in Locaiton C

4.  Dedicated MPLS to Locaiton A

5.  FW in Location A

6.  LAN Switch in Locaiton A

..... never gets delivered to MM in Locaiton A.

 

Hence, it appears that there is a problem with the IPSec tunnel on the MM's.  That is why the port 4500 Packets are not returning from Locaiton A.  It was said that the MM must have some problems with the Locaiton C MDs, after the communicaiton downage last week.  How can we fix this?

 

We have tried to change the settings on the MD side.  We have doens a write erase and have changed the PSKwithMAC settings to PSKwithIP settings on the MD side.  Then cahnged it accordingl on the MM Web user interface.  The key piece of information is that the packets are passing through the Location A firewall and then to the Main LAN Switch.... it is not being receved my the MM.  What can be done on the MM side to clear up any settings for teh IP Sec communications?

 

I am afraid that I wll need to do another write erase on both controllers this time and use a different IP address for each controller?

 

Any ideas?  Theoretically speaking this might happen at any locatin if there is a bad connection.  We will not be able to change teh IP addresses so easily if the Aruba Wireless is fully in production.

---

The reason the MDs stopped talking to the MMs was becuase the MD's uplink port (0/0/0) was set in trunk mode.  The cross connecting switch port was an access port.

 

That is why the show datapath sessio table output showed 'local' instead of port 0/0/0 for its 'Destination' field.

 

After changing the LAN switch port setting to trunk the communicatio to the MM was restored.  

Hi,

 

I am having a similar issue - weirdly after a firewall reload last night the controllers at one of our sites is now unreachable from Mobility Master - however the controllers are still online at the site and acting as they should

 

A show datapath session | include 4500 on the controller is s howing the connection betwee the MM and the controller itself as local however on other sites it is showing as port 0/0/0

 

The port is still an access port as in line with all other sites and this is the only site having the problem.

 

Is there a way to try and force the MM and the MD to communicate again and establish the tunnel?

 

Thanks

Tried everything to get the connection back up in MM, traced firewall rules, rebooted MMs and nothing.

Finally i rebooted my secondary controller on the site and it re-established the connection with MM - strange.  Will have to do the primary one out of hours tonight.  No idea why it dropped the connection and did not re-establish.

I have the same problem.

 

I already tried to restart MM and the WLCs

 

You also reload the firewall, as  I did. Do you also have a Palo Alto and version 8.1.10 installed?

 

 

I have seen this problem agian with other controllers and to fix the problem one neeed to:

 

Basically one needs to remove the IP Sec tunnel configuraiton on both sides (MD local) and teh M Side.  Then re-add it.

 

This will reuqire to use the local-config enable option that Aruba SUpport insists that you call them to do.

 

<1. The trunk fix did not work this time.  Call Aurba support and have them remove the MM IPsec tunnel from the local controller that is not conencting.>  It should never have fixed it.  It is probably something else  Either the trunk works and passes traffic or doesn't.  It doesn't just fail randomly. 

<

2.  The above proves that there was some kind of corruption with the IPsec connecting to the Mobility Master V.I.P.

     a.  At this point remove the IP Sec tunnel from the local configuration on the MD again.>  There is no proof that there is corruption.  There are many envirionments where this works and there is no issue moving forward for months.  There must be something specific with your network or configuration that is creating this issue.

 

 

There are some issues in your posts

 

Your first solution was to make the opposite side a trunk instead of an access port.  That solution either works or does not work, it does not work and then fails, so it is not a solution.

 

Your second issue is that you are attempting to make an ipsec connection through a firewall to a VRRP.  In many instances this doesn't work or doesn't work consistently depending on the firewall configuration.  If the Master of the VRRP is fairly consistent, I would try to connect  it to the ip address of the master, and not the VRRP to see if the ipsec tunnel stays up.  Support used local mode to change your MD controller, because the MD can only really be changed through an MM configuration push.  The drawback of changing through local mode is that you have to also change the specific ip address on the MM, otherwise when you change the MD independently, when it connects back to the MM, the MM will push the old config to the MD and you will be back where you started.  Troubleshooting in this manner is a very resource-intensive method of testing/configuration.  TAC cannot see everything that is going on in your network, so you might want to hire a professional to understand what is going wrong.

 

There are many reasons why and MM will not stay connected to an MD, I just outlined possibly why yours is having issues based on the limited information in this thread.

Interesting idea to have the IPsec tunnel use the Master Virtual COntorller IP address, instead of the V.RRP of the MMs.  That might be problematic if th master goes off-liine for whatever reason.

 

The Aruba workshop I w enet to provided instructions and specifically said to use the VRRP; but, that was for an internal local network. I will update the case to see if the removal of the IPsec on teh MD and teh MM and then the re-adding of it will fix the problem again.  These are now with 2 separate new controllers.

 

And we are using a Palo Alto Firewall; but, not with teh same version.  There migth indeed be something on our network that is not working well for the Aruba Controllers.

The VRRP works just fine with MMs, if they are not separated from the MD by a firewall.  Using it to the literal ip address is just an idea to troubleshoot a possible issue.

I have already called Aruba and the just opened a ticket. They are still gathering informations.

I realized, that I have a route to my MM because of the tunnel, but the tunnel is down.

So I guess, there is a really a problem with the IPsec tunnel and I believe the controller is not able to reestablish because the route to the tunnel is still there, because he sends all packets in the tunnel, but the tunnel is actually not there so they are getting lost. I don't know why a restart does not work, but can someone explain me how to reconfigure the IPSec tunnel, or how to delete this route?

 

Thank you for your hints so far

The route and tunnel are created automatically when you added the MD to the MM and it cannot be deleted.  The route is used for the MD to talk to the MM when they need to exchange informaton securely, but for establishing the tunnel, the tunnel itself does not need to be up.

 

Did the tunnel ever work?  When you type "show switches" on the MM do you see the MD?

The tunnel did work fine for about a year. 

If I do show switches, I just see my local WLCs and the MM himself, not the both WLCs with the problems.

What is between the locals and the MM?

Try typing "show datapath session table <ip address of wlc>" on the MM to see if any traffic is being sent from the WLC.  Also type "show log security all | include <ip address of wlc>" to see if there are any errors.

---

@Cullmann wrote:

The tunnel did work fine for about a year. 

If I do show switches, I just see my local WLCs and the MM himself, not the both WLCs with the problems.

---

When you issue the command "show ip route" you should see an IPSECmap route.

 

Check the ipsecmap for the problematic wlc's using the command " show crypto-local ipsec-map <name of the ipsec-map> " 

 

In the above output, what is the peer-ip address? Is it the intended IP address to form an ipsec connection with?

 

--Give Kudos: found something helpful, important, or cool? Click Kudos Star in a post.
--Problem Solved? Click "Accepted Solution" in a post.

 

Regardign this IPsec tunnel problem.. This is what has worked for me lately.

 

1.  I could not remove the masterip address from the MD and then readd it.  I needed to use a write erase instead.

 

2.  When I entered  the write erase using the same MM-VRRP Address for the master IP address, that was tehre before, so I just re-entered it. That did not work, what did work was to put the MM-Physical IP address as the masterip IP address when I tried a erite erase later.

 

3.  That connected up right away; port 4500 was communicating on both sides.  So, for some reason the IPsec tunnel using the MM-VRRP was corrupted but we want the MM-VRRP to be used by the local controller for the masterip not the MM-IP address configured.  So after I verified that the IPsec tunnel worked correctly with the MD to the MM while using the MM's Physical IP address I ran an write erase again and this time put the MM-VRRP.  That now worked.  

 

It seems as if that creating a new IPsec tunnel re-sets or removes the previous IPsec tunnell that was used for a controller, that was hosed up?  Can anyone else provide a better explanation?

Thank you all for your support.

 

I tried all that stuff but did not work for me. We captured all the packets and realized, that they were lost on the restarted firewall (Palo Alto)

 

After some research we found a current session for the controller in the session browser, but with wrong assigned zones (should have been untrust to trust, but it was untrust to untrust) we killed this session in the session browser, they came new up with the correct zones and the firewall policy which was supposed to allow this tunnel snatched again.

So since them, everything works fine. Hope this helps someone else if he run into the same problem.

 

Thank you very much for your support

Mr. Cullman,

 

I am curious.  Is your MD and MM connected through a dedicated remote connection VPN link?  Or how?  I am just curious why you had untrust to untrust?

 

It is dedicated through a MPLS line. And the LAN site is the trusted site and the WAN side is untrusted site, this is how it should be.

 

But for some reason, which we unfortunatelly don't know, the firewall assigned LAN and WAN as untrusted. It was only for this session, all other sessions were assigned correctly. But because of this wrong assignment, our policy did not work. After deleting this session, the new session were assigned to trust and untrust and everything worked fine.

Interesting, 

 

I shared that inforamtion with our Firewall Administrator and it appears to have soleve the problem with the remaining controllers that I have not performed a write erase on.  I will monitor to maek sure that the problem does not come back.

 

But for now it appears that Cullman's inforamtion has worked for us as well.  That is a lot easier and quicker than performing 2 x write erases to get the MD's to tal to teh MM again.