aarond
do you have access to the console of the AP ? if so, use ctrl+esc+k and go into /tmp, and look at the output in rapper.txt, rapper_brief.txt, rapper_counter.txt (these last two are only on recent code), and also sapd_debug_log.
Is this cpsec cap or rap ? either way, the ipsec logs are in rapper.txt , it has debug per packet, you can check for the lines that start like this:
#SEND nnn bytes to <lms ip>
#RECV nnn bytes from <lms ip>
which might give you some clue about the point at which it's getting stuck and/or the sizes of packets which are making it through.
you can also enable corresponding 'logging level debugging security' on the controller to see what point it gets up to there too.
regards
-jeff