Two things at first glance;
1. You aren't utilising the Role Mapping in your enforcement so it is not needed in this scenario.
2. Try using the FQDN of the AD group membership in your enforcement profile. E.g. CN=group,DC=company,DC=com.
What is strange however is that your users are still being allowed access when a reject is being applied?!? Can you post a screenshot of the "Summary" tab for the failed attempt?
On the Guest side, you should have a translation under your Operator settings that should match admin_privileges = LobbyAdmin and set the correct profile.