Seems this is a bug with ArubaOS. You can only use the IP address that is on the interface where the sitch traffic comes in. And as that also needs to be controller IP address, it is not possible to use two switch networks that are behind different interfaces/VLANs. Also using loopback address is not supported with dynamic segmentation.
I would've thought that someone else also has multiple interfaces on the controller or would've tried to use dynamic segmentation with loopback IP.
This should be fixed in 8.5.0.10 but that'll take few more weeks to be released...