Wireless Access

Very strange issue, this morning the Guest network could not reach Google.com or any of it's owned services (like YouTube). However, we can browse to many other websites, both 443 and 80 with no issues. I plugged into the switch the controller connects to, and can get to Google no problem in Guest VLAN, so I know it's the controller. I did a "show datapath session table" for my client IP and when I browse to google.com, I get a ton of denies. Our ACLs specifically allow HTTP/HTTPS for any location. We do have a deny statement above, but it blocks access to our 10.0.0.0/8 network, while Google is responding with 172.217.x.x. Image attached.

 

Any ideas how I can unblock this traffic? I've tried an Allow-All ACL at the top of my user-role, but still blocked.

 

Thanks.

You should type "show acl hits" over and over again to see what possibly is being hit.  

That's not a bad idea, but we have a lot of clients ~ 10K, so I'm afraid it will be constantly changing and I don't know exactly which one is from me testing. When you see the Deny flag in the datapath session table, does that mean the controller denied the traffic via it's stateful firewall or that no return traffic was found so it knows the traffic got blocked somewhere, but not necessarily the controller?

Deny is an ACL.  It could be an ACL on a port.

I checked the user-role, I checked the vlan interface for an access-group and I also checked the port (GE0/0/0) for any access lists either. I cannot find a reason for this traffic being denied. The only thing I can think of is if the controller's firewall has some sort of deny or something happened which caused it to dynamically block traffic toward a specific domain, but I've never heard of that happening before.

This is a good time to open a TAC case.

TAC caes is open in case we need further troubleshooting. Interesting enough, it did start working briefly for about 30 seconds or so yesterday evening, maybe it does have something to do with the user-table entries. I will know for sure in about 3 hours.

I ran into a similar issue last year with our online class website - after verifying my two co-workers (both gone for the week) didn't change anything via audit-trail, I immediately called TAC, but by time got on them on phone - issue resolved itself. Next morning, problem returned, got TAC on phone, and he found the problem within a couple minutes. The IP Address of our course site had entered the user-table and been given the "logon" role. If you do a show user-table for the Google IP Address when you're experiencing denies and find an entry - that would be the problem - I found they following video on the airhead forums awhile back - https://www.youtube.com/watch?v=HMIQwok5r1o


#AirheadsMobile

@cbjohns that was a great tip, I just checked the user-table and didn't find the Google address space, but I did find our DNS servers for the guest network in there, and they're session length was about the time I noticed the issue starting. I've deleted them from the user-table and will test again when I'm onsite.

 

Also, that is an excellent video for the validuser acl, actually made by a co-worker :-). I will have to look into ours and make sure it's setup properly.

 

I'll update this post when I get a chance to test.

 

Thanks!

Michael Haring,

 

Make sure you get the logs.tar as soon as it happens so that TAC can observe the state of the controller when you are having your problem.

I provided the logs.tar file and we also took packet captures, as well as the sessions logs and that was all provided to TAC.

 

Also, after deleting those invalid user-entries, the network now seems to be working normally. I've tested myself and had others verify as well, and it looks as if we are good to go.

 

Thank you both for the help, the AirHeads community has been a great outlet for solutions since I started using it. Always find great solutions from awesome users!

---

wrote:

I provided the logs.tar file and we also took packet captures, as well as the sessions logs and that was all provided to TAC.

 

Also, after deleting those invalid user-entries, the network now seems to be working normally. I've tested myself and had others verify as well, and it looks as if we are good to go.

 

Thank you both for the help, the AirHeads community has been a great outlet for solutions since I started using it. Always find great solutions from awesome users!

---

Glad to hear! I have to give credit to TAC as well. I spent that evening setting up a test AP for our help-desk (issue was seen only on one controller), stayed up checking for denies every 5 minutes till about 11PM at night, went to sleep, came in to work...broken again...denied again. Got TAC escalated while problem was caught in the act. One of the things I've appreciated from the escalation team - they won't let me off the phone till I truly understand "why it was behaving that way". :-)