Honestly, that's the way it should work. Every device on the high security side (inside of the firewall), should be able to initate or reply to any connection outbounds. You are already restricting the inbounds traffic and that is what counts.
EDIT: responses to an inbounds UDP 4500 connection could have a random source port, so you cannot really plan on what port would be used over 1024. Allowing the controller to answer on any port is the right thing to do.