Unfortunately, you can't break those groups into smaller chunks. It is ASIC's hardware design to use groups of 4 ports for certain features. But you are not obliged to have all 8 ports (4 on each stack member) wired, just configure all of them for IRF and wire as many ports as you wish - 1, 2, 3 or all 4. IRF will adapt and will use only active physical ports.
So in other words you can configure ports 1-4 on each switch for IRF, as:
Slot 1:
irf-port 1/1
port group interface TenGigE1/0/1
port group interface TenGigE1/0/2
port group interface TenGigE1/0/3
port group interface TenGigE1/0/4
Slot 2:
irf-port 2/2
port group interface TenGigE2/0/1
port group interface TenGigE2/0/2
port group interface TenGigE2/0/3
port group interface TenGigE2/0/4
and then you can wire up only XGE1/0/1,XGE1/0/2 on Slot 1 to XGE2/0/1, XGE2/0/2 on Slot 2 and IRF will work fine over those two wires. No issues with that. It's only one limitation - you won't be able to use the unconnected ports XGE*/0/3, XGE*/0/4 for normal traffic since they are configured as IRF and will work as IRF ports only.