Hello @dewced
Assigning different VLANs in the same SSID is possible for 802.1x authentication and mac-authentication. The RADIUS server should return the standard RADIUS attributes Tunnel-Type, Tunnel-Medium-Type, ,Tunnel-group-ID. On the controller you have to make sure that the WLAN-ESS interface is configured as hybrid port and you have the mac-vlan feature enabled.
It is a bit more complicated with captive portal. Captive portal feature on this controller is enabled on a VLAN interface and not on an SSID. So the user has to be already assigned to a VLAN before it can be redirected to a captive portal. Usually you can use mac-authentication in combination with captive portal for more flexibility. When the user connects to the SSID, mac-authentication happens and if the MAC is unknown the user is assigned first to a VLAN with portal authentication. This can be done either using the mac-authentication guest-vlan feature of the controller or using VLAN attributes from the RADIUS server. Once redirected to the portal page if the user supplies correct credentials the MAC address is marked as known and the RADIUS server triggers CoA disconnect of the user. The user does a new mac-authentication which this time succeeds and a new VLAN can be assigned.
So this is the basic idea how it should work, there are some details depending on which portal solution will be used. Several years ago the following technical note was released: UWW & ClearPass. How to configure Unified Wireless with ClearPass
https://www.hpe.com/psnow/doc/a00100376en_us
It contains example configuration for 802.1x, mac-authentication and guest access with mac-caching. The controller examples are ensuring compatibility with ClearPass but I think the configuration of 802.1x and mac-authentication is valid for every RADIUS server. The guest solution is more specific to ClearPass.
Please note that this document is from 2016 and I dont know if it was updated. The Unified controllers are end of sales since 2017 while ClearPass had many new versions in the meantime. So some things may not be valid any longer.
Regarding CoA it is good to know that this controller has very limited support. It supports CoA request with 2 attributes only, Termination action with value 0 and session-timeout.