Looking into the datapath has led me to the workaround, but still don't understand the cause
Findings
1. datapath session table showed Deny flag all over the place
2. acl hits showed as below (everything is being blocked by "sdn-acl") (I assume it is blocked/dropped, I did not see a listing of what ofaction 14, 11 or 1 means)
3. sdn-acl is not used anywhere and also has zero rules configured in it
4. The only acl applied was per-session global-sacl on Port Channel (global-sacl also has zero rules)
5. unable to change config on port channel to 'not defined' (error message: "deletion will not occur as the item is inherited", but unable to change it even at the top level)
6a. changed per-session acl on port channel to allowall
6b. all came to life, got VRRP, got lc-cluster, got remote reachability
ACL Hits during issue:
#show acl hits
User Role ACL Hits
------------------
Role Policy Src Dst Service/Application Action Dest/Opcode New Hits Total Hits Index Ipv4/Ipv6
---- ------ --- --- ------------------- ------ ----------- -------- ---------- ----- ---------
Port Based Session/Route ACL
----------------------------
Policy Src Dst Service/Application Action Dest/Opcode New Hits Total Hits Index Ipv4/Ipv6
------ --- --- ------------------- ------ ----------- -------- ---------- ----- ---------
sdn-acl any any any ofaction 14 144597 144597 8353 ipv4
sdn-acl any any 17 0-65535 ofaction 11 2 2 8373 ipv4
sdn-acl any any any ofaction 1 2184305 2184305 8375 ipv4
Port ACL Hits
-------------
ACL ACE New Hits Total Hits Index Ipv4/Ipv6
--- --- -------- ---------- ----- ---------
ACL hits after workaround applied:
Port Based Session/Route ACL
----------------------------
Policy Src Dst Service/Application Action Dest/Opcode New Hits Total Hits Index Ipv4/Ipv6
------ --- --- ------------------- ------ ----------- -------- ---------- ----- ---------
sdn-acl any any any ofaction 14 274 144883 8353 ipv4
sdn-acl any any 17 0-65535 ofaction 11 0 2 8373 ipv4
sdn-acl any any any ofaction 1 5176 2189731 8375 ipv4
allowall any any any permit 256 256 7895 ipv4