I've somewhat figured out my problem but it doesn't necessarily make sense to me how the controller is handling this.
The default group (in ACS) that my test user was in was not setup to return any value for "Filter-ID" which to me means the authentication would fail since I thought it would be looking for the "allowaccess" attribute. It seems that if there is not a attribute returned the user is allowed access instead of denied.
To fix it I set it up to the default group to return a "denyaccess" for Filter-ID and then added a second server rule looking for that and assigning the denyall role.