If I have understood what you're saying correctly...
1. You have ports 0 & 1 connected from the controller to the switch.
2. Port 0 connects like an "access" port on vlan 1 (i.e. untagged).
3. Port 1 connects like a "trunk" port (i.e. other tagged vlans), AND you've left vlan 1 untagged on this port also.
4. The controller IP is on VLAN 1.
If this is all true, it probably explains why you're seeing the issue you describe. I would have expected in this scenario one of the two ports to block via STP. Can you check the STP status on your controller and switch to which it attaches?
Regardless, your setup isn't ideal. I suspect your traffic is lost due to switch-controller interaction in terms of the STP, source-mac learning or something similar.
I would recommend you either need to:
1. Use a port-channel/ether-channel between the controller and switch, using multiple ports carry the same set of VLAN traffic (including VLAN 1).
2. Take VLAN 1 off the trunk port (i.e. port 1) with a VLAN allowed list. Note that you should check your STP status first!
For a simple test to prove my theory (assuming vlan 1 untagged is allowed on your controller port 1, and the switch port to which it attaches), just unplug your controller port 0. It should work fine like that.