When a device authenticates successfully using dot1x, you flag it by setting an attribute in the Endpoint Repository (e.g. Dot1xEnabled = True)
Then in your MAC Auth service for the non-dot1x SSID, you check that this attribute does not exist.
Other than that I don't see how you can do it, since your non-dot1x service is not capable of detecting dot1x devices (by definition).