Wireless Access

I recently deployed several ANT-2x2-5314 antennas to several MSR-4000's so that we could make use of DFS channels for some backhaul links to eliminate channel reuse/interference. 

The deployment is at a research factility in a remote area in Colorado.  The nearest town is about 4 miles away.  Since making use of the DFS channels, we are seeing some connections drop and not reconnect for 20-30 mintues.  The radios that should be connected will be able to see each other, but they will be unlinked due to 'peer gateway cost higher'.  There's no other radio for the far-end to connect to, there's no blacklists, the near radio will not be at it's peer limit, the RSSI should meet the minimum required, etc.  There's no reason that I can see for the radios to not make a connection.

When this occurs, the following message is noted in the logs:

Jun 20 01:50:49 MSR4000 PoE: Interface:Warning:core2:Mon Jun 20 01:50:49 2016 :Radio interface wifi1 Channel has changed to wireless mode 802.11na channel 116

Nobody is changing the mode of the interfaces as the configuration has been set for many weeks now, so the radio should already be in that mode.  I believe the connection fails at the time this is logged and it will be some time after this message before the radio will reconnect.

The radio can run for weeks without issue, then we will get this several times throughout the day.  So far we have seen this on all of our DFS links on various channels (100, 108, 116, 136), but we've never seen this on a non-DFS channel.

When I check the channels on the radios having this issue, the radar counters will be zero.  The signal quality for these links are typicaly in the 80-83% range.

Any suggestions on other things to check would be great.  We checked with our partner and they had never heard of this warning before.

So the radios are changing due to DFS events? Are you near any doppler or weather or radar? Where **exactly** are these APs?

I don't have any reason to believe there are DFS events as the channel selector screen that shows how many radar events there have been shows 0.  I would assume that a DFS event would log something indicating that there was a DFS event?!

Location:

https://goo.gl/maps/2QSVZxwTRKp

The research facility is the townsite of Gothic, CO.  The research facility isn't using any kind of radar.  Their area of research is biological studies.  Other than the research facility (70+ buildings, most are small cabins built in early 1900's), there's no significant structures for miles.  There are a few small cabins outside of Gothic, most are off the grid.  Law enforcement is about 4 miles away and there's a mountain between Gothic and where police radar would be used.  To clarify on why law enforcement would be 4 miles away, you are passing through National Forest to get to Gothic and Gothic is completely surrounded by National Forest.  Law enforcement and other emergency services can only access from the southeast.

The nearest is Crested Butte, CO.  Year-round population of 1800.  There is a wireless ISP there that I know makes use of Ubiquity equipment and uses DFS channels in the area without issue. They are also providing internet service to this research on channel 64 (5320Mhz) and that connection rarely has any issues.  I have confirmed with the owner of the ISP that they are not using any other channels in the area nor do I see any other channel usage.  Everything else the ISP has deployed is on the other side of the mountain.

This is a REMOTE area.  You don't end up in Crested Butte, CO unless that's your destination and you certainly don't pass-through Gothic, CO while traveling elsewhere.

DFS is subject to transient events, which could be triggered by an overflight of a plane/aircraft with their radar on, or even simply a false positive. If you need absolute stability on a mesh link outside, you should use one of the UNII3 channels. You can open a TAC case as well to see if they can pin down or get some logs to identify, but from the sounds of it, it sounds like either MeshOS decided to change channels (assuming you have multiple channels enabled).

I am upgrading my MSR cluster to 4.7.0.3 to look at some of the settings, but unfortunately, DFS-enabled radios are VERY sensitive to any DFS triggers. Unfortunately pinning down DFS related issues is very tedious. If you need to work the issue, TAC will be the best bet.

btw I live in Erie, I am roughly familiar with the area :) And if you are on or near Gothic, that means higher elevation which means you get to pick up more interference than you would down at lower elevations when aircraft fly over.

Jerrod,

It is definitely at higher elevation (9600'), but also in a narrow/deep valley surrounded by 12k,13k,14k mountains.

The local WiSP in Crested uses DFS channels for backhaul on Ubiquity equipment without issues.  Don't know if it isn't as sensitive to radar or what.. I know they have an AP much higher than ours near a mountain peak in Irwin, CO using a DFS channel almost 30 miles to Gunnison, CO.

Let me know if you're ever coming out to Crested Butte!

4.7.0.3 is out, but not public. Check your PM. It's about as good as I can do. You can setup a syslog server and point the 4Ks to it at debug to see if you capture. That is a higher number of events than I would expect to see, but again, it's hard to know. DFS is *very* tricky outdoors, always has been and always will be, and disruptions will be expected. Wish I had better news. Maybe 4.7.0.3 has some tweaks (though the RNs have nothing noted). 

4.7.0.3 DOES have a bug fix in it, though I cannot look at the bug details to know if it's related to your issue or not. However, as MeshOS is in maintenance mode, it won't be getting a lot of attention.

11786 Symptom: Altering MSR2000, MSR4000 and MST200 channel set to non DFS channel for US

operation.

Scenario: This issue is seen in MSR2000, MSR4000 and MST200 routers running MeshOS

software.

They don't have multiple channels enabled on a single radio interface.  There are grouping of AP's setup to connect back to 2 'core' AP's.  Each grouping is setup to connect to a specific radio/channel on one of the 'core' AP's.

When they go down it is usally for 20-30 minutes, but sometimes less.  I happened to catch an occurence last week and manually restart the interface on the impacted 'core' AP and everything reconnected within 60-90 seconds.  If the interruptions were only that long I don't think anyone would complain.

If there's anything I can do to enable more logging in relation to DFS, I would very much appreciate any guidance.  There's nothing in the logs when these events are occuring, except what was noted in the original post.  Would a radar event likely impact one channel or several channels?

I'm also dumbstruck by the radar counts all showing 0.

This AP has an uptime of 17 days and had 4 incidents today:
1:50 am
2:50 am
9:33 am
10:33 am

The original design was using UNI3 channels with 20Mhz of seperation, meaning only 149, 157, and 165 were used.  Would we have success using all 5 UNI3 channels?  Our partner said using channels with no seperation was not a recommeded configuration. 

The two main 'core' MSR-4000's are on the same building about 50 yards apart and there are 13 AP's directly connecting to these two AP's.  There were 3 'core' MSR-4000's on that same buidling within about 100yards of each other using 149, 157, and 165 on each AP and every evening when the scientists and students would return to their cabins the signal quality at the 3-cores would collapse and the network would basically be unusable all evening.  Using DFS channels there's no longer any channel re-use on the core AP's , so signal quality stays solid and the network performs great except for these occasional interruptions.

Is 4.7.0.3 public?

I'm working to get these radios under a support contract.  This is a non-profit and money is tight.

Here's the output of 'show mesh channel-information', there's no sign of radar interference and the noise floor is in good shape.

core2# show mesh channel-information dot11radio 0
Column name abbreviation
  NF: channel's noise floor;
  UTIL: channel's total utilization;
  WIFI: channel's WIFI utilization;
  INTF: channel's interference utilization;
  INTF-L: interference level, for detection and avoidance;
  NGH-CNT: channel's backhaul neighbor count/connecting to me;
  ADJ-UTIL: channel's adjusted utility, for channel selecting;
  FLAGS: channel flags;
    A - Allowed channel;
    D - DFS channel;
    W - Weather radar channel;
    R - Radar detected;
    I - Heavy interference channel;
    O - Overlap with other radio;
    C - Candidate channel;
Radio 0 channel list:
Channel NF   UTIL WIFI INTF INTF-L NGH-CNT ADJ-UTIL FLAGS
60      -      -    -    -  -       0/0       -      DC
64      -      -    -    -  -       0/0       -      DC
100     -96   8%   8%   0%  0       3/3      0%      ADC
104     -      -    -    -  -       0/0       -      DC
108     -      -    -    -  -       0/0       -      DC
112     -      -    -    -  -       0/0       -      DC
116     -      -    -    -  -       0/0       -      DO
132     -      -    -    -  -       0/0       -      DC
136     -      -    -    -  -       0/0       -      DC
140     -      -    -    -  -       0/0       -      DC
149     -96   4%   3%   1%  1       0/0      2%      O
153     -      -    -    -  -       0/0       -      C
157     -      -    -    -  -       0/0       -      C
161     -      -    -    -  -       0/0       -      C
165     -      -    -    -  -       0/0       -      C

You are statically setting radio channel correct? You could also try to enable 2-3 channels per RADIO so that if one triggers, it can switch to something else. If you lock a radio at 100 and then DFS kicks, the radio will have to shut down completely. With MSR (and I haven't tested this and won't be able to anyway), maybe by having an alternative channel (remember to enable those same channels on their children as well), if it triggers on 100, it can move to something else like 108.

I'll give the multiple channels a try on a group and see how that goes.  The DFS channels give us some room to do that.

Any comments/advice on using 20Mhz of seperation on channels?  That greatly limits available channels.

If you are doing three 5Ghz in the same chassis, 40Mhz separation is *minimum*, more is better.