In the service rules for mac, do you have a line called:
Connection | Client-Mac-Address | EQUALS | %{Radius:IETF:User-Name}
This should prevent dot1x clients to hit the mac-auth rule, you can then place the mac rule above dot1x.
Or you can create a more specific condition in the dot1x service to exclude mac connections.