My understanding is that every switch in the IRF stack needs to have an IP address registered on the MAD BFD VLAN. Here's an example from my two-switch 5500 stack:
interface Vlan-interface555
description MAD BFD for IRF monitoring
ip address 10.5.55.254 255.255.255.0
mad ip address 10.5.55.1 255.255.255.0 member 1
mad ip address 10.5.55.2 255.255.255.0 member 2
I believe you would extend this by simply adding:
mad ip address 10.5.55.3 255.255.255.0 member 3
mad ip address 10.5.55.4 255.255.255.0 member 4
And then assign two ports in VLAN 555 to each switch, chaining from one to another in a ring, just like your IRF links.
Disclaimer: i have not done this - my IRF stacks have only ever had 2 switches each. :-)