You can do it.
But be aware that each host will then not have real VLAN isolation.
As an example, if you have a baby switch connected to the Aruba CX, and your clients are connected to that switch, they will all see broadcasts for all the clients on the switch. As technically they will only be part of the assigned VLAN after the traffic enters the CX port. Before that, they are sharing the same L2.
It is also quite trivial to piggy back into a VLAN being used by another client connected to the baby switch.